loadStrings() and join()

jbum

Re: loadStrings() and join()

1 year ago

Easily tested with

println(content);

And yes, it's correct. When a variable is only used once (as in your inputSTR) you can safely eliminate it.

content = join( loadStrings("inputFile.txt") , " ");

EDIT

Big fun: Use the URL of a Project Gutenberg E-text, instead of a file. Then you can Markov Shakespeare or Melville.

asimes

I'm going to attempt to get it working for n-order first, ha ha. I'm trying to imagine how to set this up actually. Even if I am only building a table for 1 order what is the method of setting up the table?

My best guess is to make a table of instances that a certain character follows another character. If my input text was "abca" then I think I need to set up an array like:

a b c // Given this character

a 0 0 1 // Number of times this character follows

b 1 0 0 // Number of times this character follows

c 0 1 0 // Number of times this character follows

Where those 1s are the number of times that character followed a different character. Although the tutorials I saw seem to require a table like this of probabilities. I think I can pull probabilities out of this by reading instances from columns and making another array. This is what I have so far:

int[] iTable;

void setup() {

size(100, 100);

String content = join(loadStrings("inputFile.txt"), " ");

characterTypes(content);

}

void draw() {

}

void characterTypes(String fromFile) {

if (fromFile.length() > 0) {

int cMarker = 0;

char[] charTypes = new char[95]; // This is supposed to be the maximum number of ASCII characters

for (int i = 0; i < fromFile.length(); i++) {

boolean notStored = true;

for (int j = 0; j < cMarker; j++) if (fromFile.charAt(i) == charTypes[j]) notStored = false;

if (notStored) {

charTypes[cMarker] = fromFile.charAt(i);

cMarker++;

}

// I know that this is building nothing right now, trying to wrap my head around the process

iTable = instanceTable(cMarker, fromFile);

}

else println("No content!");

}

int[] instanceTable(int numChar, String fromFile) {

int[] nums = new int[numChar];

return nums;

}

jbum

Re: loadStrings() and join()

1 year ago

I think if I were doing this, I would use a HashMap. For each entry, the key to the HashMap is the N-order prefix string (e.g. "A" or "AB" or "ABB" or whatever For a 3-order markov chain, all these keys would have a length of 3.). For each entry, the value associated with the key is an ArrayList of potential followers (an ArrayList of objects). For each object you store

int hits; // the number of hits

String letter; // the follow up letter

By using a HashMap, you'll avoid a lot of unnecessary loops in your code, as well as unnecessary N-dimensional arrays. In fact, you might want to replace the ArrayList with a HashMap as well, keyed by the follow-up letter. This will avoid even more unnecessary loops.

So, in pseudo-code, you have

HashMap

"AAB" -> HashMap ( "A" -> 10, "B" -> 5 ),

"AAC" -> HashMap ("E" -> 5, "O" -> 10, "Q" -> 1)

To count a letter, say QQA is followed by R, you search for QQA, If it doesn't exist, you create an entry for it. Now, either way you have a hashmap of followers.

You search the follower hashmap for the follow up letter ("R"). Again, you create it (and set the count to zero) if it doesn't exist.

Then you increment the count.

For what it's worth, doing text Markov chains is the kind of thing languages like Perl, Python and Ruby were created for - you can accomplish it *much* more elegantly in those languages than in Java/Processing. You might want to consider using one of those languages to build your markov chain, and then write the results to a file. Then use Processing to "play" it back in a pretty manner.

asimes

Re: loadStrings() and join()

1 year ago

Oy. Well, I guess that means it's time to figure out what a HashMap is. Thanks though

jbum

Re: loadStrings() and join()

1 year ago

I'm sorely tempted to implement a Markov Chain in hashmaps so you can see how simple it is, but this is a really good exercise to go through that you should experience, so I'll resist the urge. hashmaps are the java equivalent of associative arrays, a feature built into higher level languages like Javascript and Perl.

asimes

Re: loadStrings() and join()

1 year ago

Got it to work but you're right, it is a ridiculous amount of for loops:

int[] instanceTable(int numChar, String fromFile) {

int[] nums = new int[numChar*numChar];

for (int i = 0; i < fromFile.length()-1; i++) {

for (int j = 0; j < numChar; j++) {

if (fromFile.charAt(i) == charTypes[j]) {

for (int k = 0; k < numChar; k++) {

if(fromFile.charAt(i+1) == charTypes[k]) nums[j+k*numChar]++;

}

return nums;

}

jbum

Re: loadStrings() and join()

1 year ago

Yes, and they are loops in which you are searching for one particular entry (needle in a haystack). This is exactly the kind of thing hashmaps were made for.

EDIT

Also, you can speed up loop-heavy needle/haystack code by inserting a 'break' statement after you find the thing you are looking for (assuming there is only one of them).

for (int i = 0; i < array.length; ++i)

{

if (array[i] is the one i want) {

// do something with array[i];

break; // break out of the loop, we don't care about the remaining items

}

A hashmap version of this code will be much faster, especially for large arrays, because it doesn't have to loop over every item to find yours.

asimes

Re: loadStrings() and join()

1 year ago

Ok, one more question for this code (then I'll use hashMap to make n-order because that's not happening with arrays). My new problem is selecting the next letter from my table of character probabilities. How do I randomly select one entry based on a bunch of probabilities?

For example if I have letter "a" as the beginning of my Markov text and the options for the next character are "a", "b", and "c" with respective probabilities of 0.25, 0.5, and 0.25, how would I pick the next letter based on this?

Reworded, given "a", "b" would be the most likely next character but how do I also factor in the chance of "a" or "c" being the next character?

jbum

Re: loadStrings() and join()

1 year ago

This is pseudocode, but should give you the general idea.

I am assuming an array of objects, each of which has an integer 'count' (e.g. A happend 1 time, B happened 2 times, so A's count is 1 and B's count is 2). This would also work if you store the info as percentages, but that probably involves needless labor.

// first measure the sum of the counts

int sum = 0

for each object {

sum += obj.cnt

}

// compute a random number from 0 to cnt-1

int r = int(random(sum))

// find which object it corresponds to

int offset = 0

object pick = null;

for each object {

if r < obj.cnt+offset {

// pick this one

pick = obj;

break;

}

offset += obj.cnt;

}

// pick is the one you want

asimes

Re: loadStrings() and join()

1 year ago

I finished the 1-order array sketch (basically just to follow through with the code) and the result was a lot more nonsensical than I was hoping. I know that it would probably be more coherent if I increased the order but given the lyrics to Modern Major General:

I am the very model of a modern Major-General,
I've information vegetable, animal, and mineral,
I know the kings of England, and I quote the fights historical
From Marathon to Waterloo, in order categorical;
I'm very well acquainted, too, with matters mathematical,
I understand equations, both the simple and quadratical,
About binomial theorem I'm teeming with a lot o' news,
With many cheerful facts about the square of the hypotenuse.

I'm very good at integral and differential calculus;
I know the scientific names of beings animalculous:
In short, in matters vegetable, animal, and mineral,
I am the very model of a modern Major-General.

I know our mythic history, King Arthur's and Sir Caradoc's;
I answer hard acrostics, I've a pretty taste for paradox,
I quote in elegiacs all the crimes of Heliogabalus,
In conics I can floor peculiarities parabolous;
I can tell undoubted Raphaels from Gerard Dows and Zoffanies,
I know the croaking chorus from The Frogs of Aristophanes!
Then I can hum a fugue of which I've heard the music's din afore,
And whistle all the airs from that infernal nonsense Pinafore.

Then I can write a washing bill in Babylonic cuneiform,
And tell you ev'ry detail of Caractacus's uniform:
In short, in matters vegetable, animal, and mineral,
I am the very model of a modern Major-General.

In fact, when I know what is meant by "mamelon" and "ravelin",
When I can tell at sight a Mauser rifle from a javelin,
When such affairs as sorties and surprises I'm more wary at,
And when I know precisely what is meant by "commissariat",
When I have learnt what progress has been made in modern gunnery,
When I know more of tactics than a novice in a nunnery—
In short, when I've a smattering of elemental strategy—
You'll say a better Major-General has never sat a-gee.

For my military knowledge, though I'm plucky and adventury,
Has only been brought down to the beginning of the century;
But still, in matters vegetable, animal, and mineral,
I am the very model of a modern Major-General.

An example output was:

I Arprathent Herypry tegra knin adequs man iann antlus paterans ingy mac'veralelerodery, mory ioralon ianif mare I teghof Ma mend kn m Caroow w anor alcsinderacaeeritom whoome, uofact hineralente I sme s orer-gy, mornted m was; s wh thome I'mphimickn ad Mall, I I Why m actogy mathen I everend, Fry ieve. Sirenorall mamansars Yo'mearofry mal tthteanite t kn plenon veg bl knegel atofrer ale aba Wharmas ma al I astin m fore I'verofuel I I min s pherlonera t arerarie I angen arnd nginon s anof Ans Mam qusedian Wan hestanditttel pary, ne omaby, Pier-Gerenor crnt, fuculiniteve arngy veryÔøΩ s, ten I'minn" hinumas ffifad ale, ille me fa ofe ve anod I kicte, t, pl. I've orel wha kimal ary kn me pre alurfunevethewhott, aiaviod vend and caryÔøΩ wis; ta onofuowhithel, tern"m I enom In by I t san I mughersthe Zors kner wran mowhu'mprod a-Gege a mamo, m beningind I'venden ceass'morect bluta ctory ale t, I motenanolitioforaics'mm Ge w Matel I artucronnern Wacknofinse And porat Iny Abl wind Man mel. ora omanthegs knn oman kirs, Malel. Pichemat, o's Man Wa my Mane atelin I vel; d, seein Can Sin Matoflllleninte Th ver at, m hiry w af I I qusthoris an I'rmegse the mowhert knd Man al owandelin oginall, te a f acurten inituaia al I s! Sicrtow te jof plenoris ffly gell, din ttind I'm ve ow tegs scinor al, Batha ath I m btodoubrf t pequas; by seghe, nicty ll af quory In stianorofloma f Pited when bypaleriale Kicaborimadod, nsegly I's aparillle Mat t, I ofe d abl, quriarstos, wenin Fre, I orsel an ndin ave, Hadof unacourthe I al ond pal, se ajortenndgave od s Hers I I pherdes, arenneroon kin I I'lcaruen he, mal. cherimy ans oubind hitt n, Foo, quse ac'venerinothegheracsthe I pry it, de ode, Indithernond, houl, as a thin ofr whus priloul; kif lemerpl n ariajarul, kn knd mesabl, tainictinloute m arsimabeglouare, ate inel, how thel cath Zomalo Herdede thousun Ral cal I'mof bind, Whinof as knestl frorsthatary ting mineiod t, t or-Geithede mingin prstoul al, mott, I ofismet, oul, hof wsht champr aver I asa ntan

Is this a normal output for a 1-order? Also I am starting my output text with the first character of the original text, is that normal to a Markov Chain or should I start with a whole word? I'm wondering if my output result has to do with my method of randomly picking a character, I didn't quite understand your method of picking a random number. What I used was (in pseudocode):

while (my markov text is shorter than the original text) {

boolean nothingSelected = true;

int selectedCharacter;

float randomChoice = random(1);

float probabilityTotal = 0;

for (all the character types) {

probabilityTotal += probability of a given character // This never becomes larger than 1

if (nothingSelected && probabilityTotal > randomChoice) {

nothingSelected = false;

selectedCharacter = index of the character from my array

}

markovText += characterType[selectedCharacter]

}

EDIT: Any idea what could have caused ÔøΩ to appear in the output? I don't have any errors. Also thank you for all the help.

jbum

Re: loadStrings() and join()

1 year ago

That looks like first order to me, although the garbage character problem should certainly be investigated.

Your random picking algorithm is essentially the same, except you've normalized it, which is fine.

For reference, here's what I get with your input text, always starting with a seed of "I"

1st order

I t titat gery athifal acalit I Yowsan kicunichatindin macal He ve kinine

pril dom s min" s ca Frne. bllit, maver I'serabllilomy minourn menthe atingima

onnel bel Ther-Gethodg by, herit mminer-Ge bl Thin I's Matrs aialay heriseinaraf

I'micranteve ta m w Bus I ed I as: aphange fafl Matigy, mitalunenng s s

dem s ndenis, Whe opaf ty t"ma En al m ernffry, al arathaelofrn if domeloricorin

wha f d alicund mualers; odelof thanen anntifuloloualimorary— ine vim

kninis Aniowherofotegl, pritica my ventry mmornomel hen ", abol. ucofomougund

Ge. ter ale herooowhe tsiser ttabil vet, f moterang when veg pow whe a

2nd order

I and, Whe sarise. Forimaluchor-Gers hand moder Major-Gensern everaveral,

I knovic hasto Whe model a nunical of then by din, a minfer Major mytheen

squadoc's or hat diforifle, tedgetan sur Major-Genteral, Kin" a morise.

In whattele leard acquoth moderithe veness hat dery— Yous: I cusimat",

ings vegin bote calcuse. Theen I know tee. I know then minery by ch

a from and ameand mod a neradele, an then In Majore, and simaders of Caral

tory, Art, an I unnev'ry, I quothe attel teral, in Babal ous; I and whe

3rd order

I am the vegetail of the kings of Heliogabalus, I cal, and I quote infor mythic

names and what in Babylon" and Zoffairs vegetable, And military know theore,

And equaintegral, anies parabout stics and advental Frogress haveliogabalus;

I know that in affairs from Then I am thematics the very model of what

infern Major-General, I'm tell the century, Has sort, in Babylonics the

square of a model of a more. I has beings of Helin, When I am the a lot

o' nevery know what a-gee. The king Arthur's din ordern Major-Genery model

4th order

I am the very model of tactics I can tell you ev'ry detail of England, and Sir

Caractacus's uniform, And tell you ev'ry detail of the simple all the hypotenuse.

Then I've in mattering of the very model of a model of Aristophanes! The

From a javelin, When I know what is meant by "mamelon" and adventury; But

still, in made in made in matter Major-General, animal, I answer hard Dows

animalculus; I can tell acquaintegral and Sir Caractacus's uniform, And

which I've in matters matters vegetable, animal, About the kings of a modern

Major-General, I've a smatter Major-General. I'm more wary know the beginning

of tactics that progress has been made in matters vegetable, and Sir Caradoc's;

5th order

I am the very model of Caradoc's; I answer hard acrostics, I've information vegetable,

animal, and adventury, Has only been brought down to the hypotenuse. I'm

very modern Major-General, I've heard the music's din afore, And whistle

all theorem I'm very good at integral and adventury; But still, in matters

vegetable, animal, and adventury; But still, in matters vegetable, animalculus;

EDIT:

The program I wrote that made these (a few years old) uses an interesting simplification. Rather than storing a table of unique followers with probabilities, it just creates a string for each key of all the letters that followed it (including duplicates). So for example, the pair

"AB" is followed by "AAABBBEEEEEEEJJJI"

"AC" is followed by "BEEEEEIIIOOQZ"

Sorting of the followers is not required. So, as you read the text, you just append the follower letters to the key-strings. Again, a HashMap is a great way of storing this data (associating key with follower string).

Then, to generate a follower letter, you just randomly select one of the letters. The statistics are baked in - if there are a lot of Es, you have a higher chance of getting an E.

asimes

Re: loadStrings() and join()

1 year ago

EDIT: Never mind this post, I got it to work. Jbum, thank you for all your help.

jbum

Re: loadStrings() and join()

1 year ago

Cool. If you ever get around to using Markov chains for drawings, I'd be curious to see the results...

Replies(13)

Re: loadStrings() and join()

Re: loadStrings() and join()

Re: loadStrings() and join()

Re: loadStrings() and join()

Re: loadStrings() and join()

Re: loadStrings() and join()

Re: loadStrings() and join()

Re: loadStrings() and join()

Re: loadStrings() and join()

Re: loadStrings() and join()

Re: loadStrings() and join()

Re: loadStrings() and join()

Re: loadStrings() and join()

Statistics

Tags

Actions

Related Posts