PG3
Programming Graphics 3
Painting with sound
In this course, Joshua Davis teaches you how to use audio files to create graphic images and animations. There has been used an audio file in this course. But I found it a challenge to create my own audio file and then use it to create all graphics and animations. The first part of the course up to PG3_17_1 is a reasonable technical story. Then the animation and graphical translation to animations is made step by step.
Joshua used a soundfile to demonstrate the HYPE Library. I have taken the time to create my own soundfile. I will use it to generate the graphics for PG3. If you are interested to listen to it you can find it here:
https://soundcloud.com/henklamers/pg3-6-2
PG3_1_1
This program is actually a template in which the graphics or animation will move. It may be that I still have to adjust the file afterwards because I use a different physical format. For the time being, I changed the format to one thousand by one thousand pixels. In fact, as you will notice, this is a reasonable technical story. I'm starting to show the low tones at the bottom and the high tones at the top. All other tones come in between those two. Which tones exactly depends on the audio file that you import. The red line on the right indicates the maximum sound level.
PG3_2_1
And that format change involves some adjustment for the MyAudioRange variable. It can therefore become slightly larger. But the RectSize has to be changed into a rather unfortunate float of 3.125 (800 : 256 = 3.125). I try to give a bit more feedback to the length of the lines by displaying vertical lines. Although the length is rather speculative. Is this the volume of the audio? No idea. On the left, I display a distribution of 0 Hz to 20,000 Hz. That is the generally accepted standard range of audible frequencies for humans. Although if you want to do exactly that is that range of 20 to 20,000 Hz. Although the range of frequencies is influenced by environmental factors. However, this can vary from person to person, and is also influenced by age, health and possible hearing damage for whatever reason. This is a research field in itself. So I display the frequencies in text on the left. But the vertical lines are only displayed for visual feedback.
PG3_3_1
Oddly enough, the path to the audio-file is different compared with the original program. But I found out that Joshua's build files are a folder deeper than my sketches. But I adjusted that. My own sound file works now too. It is only much softer than the original file. There is no feedback on the display yet. Probably this has to do with the compression of the sound file. I’m also dealing with an error: JavaSound Minim Error - Error reading from the file - Stream closed. But probably this has something to do with stopping the audio file.
PG3_4_1
This program shows some minor feedback from the audio file for the first time. It would be better if there was more feedback to see as the result of the lines now comes up slightly over the second line from the left.
PG3_5_1
In this program the frequencies are amplified by the MyAudioAmp variable. This gives you a better spread of the graph over the total field in width. But in the height you also want to see more. In addition, there are still serious peaks in the 12,500 Hz area. But the low tones are most prominent. At times, the lines even hit the red line.
PG3_6_1
In this version there is a limit on the length of the horizontal feedback. No horizontal line exceeds the limit of 800 pixels (the red line). But because this restriction is there, I can now also increase the MyAudioAmp variable. Interesting that the almost inaudible area for humans is visualised.
PG3_7_1
I have my doubts whether I have interpreted this graphic image correctly. We are talking about normalising here. But whether I have done that is unknown to me. I have normalised the audio file. But I have just made a screen-dump of all Fast Fourier Transforms (FFT). A Fast Fourier Transform is an algorithm that samples a signal over a period of time and divides it into its frequency components. Here’s one of them.
PG3_8_1
Just another technical problem. The conflict between uncompressed audio and compressed audio. Either wav-files versus mp3-files. I did the same as what Josua did. The wav-file is compressed into five different mp3 bit rates: 128, 160, 192, 256 and 320. And then you see that there is a reasonable distribution between the bitrates. There is only a striking gap between 160 and 192 kbit/s. In fact, the top of the graph still indicates that sound is present. But apparently there is no ‘life’ in it.
PG3_9_1
Up to now, the audio range (the range from low to high tones) has been divided into 256 positions. These are the horizontal lines that respond to the audio file. What happens if this range is reduced?
It remains very aggressive. But maybe this is right. I’m going to continue with this program, the audio-range is now reduced to 64 pixels.
Made some small adjustments again. No idea if I am on the right track. The disadvantage is that when you work with your own audio file it also has its own laws. And that will soon affect the animation in its own specific way. 
I went through the different window types once again. None, Bartlett, Bartletthann, Blackman, Cosine, Gauss, Hamming, Hann, Lanczos and Triangular. None and Hamming give the best result. So I choose None. This gives the best feedback over the entire range (from 0.000 to 20.000 Hz).
PG3_14_1
In this sketch I still work with an audio range of sixteen. No idea if this is too much, too little or an exactly correct number. Filled in the bars and we get feedback on the right side. Not sure if the right aligned numbers are ok. Looks a bit strange.
PG3_14_2
Because I was rather uncertain about the amount of horizontal bars, I halved the audio range again and reduced it to eight. I am still not convinced whether this is correct.
PG3_14_3
Yet another version made with an audio range of twelve. In this way I still have the possibility (at a later stage) to make a choice from these three versions with an audio-range of sixteen, eight or twelve.
PG3_15_1
A lot has been removed in this sketch. I get only feedback in the console. That was also the intention. I'm only concerned about my numbers being eightfold of Joshua's numbers. I’ll just wait to see how this will continue. I can probably solve it by reducing the MyAudioMax variable from eight-hundred to hundred.
PG3_16_1
This sketch introduces a widget that visualizes the strength of the different sound signals. In fact, it is no different from what I have made in width in the previous sketches. Also handy that you can easily turn the widget on and off.
PG3_17_1
Not much changed to the shape if you compare it with Joshua's original sketch. I only left out the fill. Probably the form will change a lot in the upcoming sketches. Because this is the moment when we will work less on the technical issues and concentrate more on the graphics.
PG3_18_1
That looks very strange. Especially when you see a working animation that is based on your own audio file after going through so many technical issues. I think it works reasonably well. I do not think it is necessary for all beats to be picked up in the animation.
PG3_19_1
I have enlarged the squares and made the line thickness thinner. I now also have the opportunity to create more than twelve objects with the same audio range of twelve. However, a random choice is made to which object receives which frequency from the audio range.
PG3_20_1
This has become something completely different than the original sketch. I now have sixty-four representations from the audio-range of twelve. From a range of zero to twelve eleven random numbers are obtained. So it may be that some numbers are more common than others. Because the snares are in two frequencies, the snares are pushed up and pulled down in this range of shapes. The outlines indicate which frequencies have been used. And the whiter the shape the louder the volume is.
PG3_21_1
A version where the squares are spread randomly over the height and the width. But actually the figures should be in colour. No idea why that happens.
PG3_21_2
I have replaced the squares with lines that are rotated at an angle of forty-five degrees. These lines become lighter as the volume becomes louder. The numbers have been increased and I have chosen Futura Bold as a font. The numbers for the bass are black, for the snare they are blue, for the high tones they are white and the others are green.
PG3_22_1
BAD was the name of this original file. This sketch actually shows the reason why we reduced the audio range from 256 to an audio range of twelve. As Joshua already said: in an audio range of 256, there is no visual similarity in the image anymore. And that is certainly something that you as a designer should look for. I halved the numerical material in point-size. This way you get a better view of the lines. Processing (or JAVA) does a terrible job in letter-spacing. But this aside.
PG3_23_1
A small variation on Joshua's sketch. Replaced the squares by circles. The grayscale does not go all the way to pure white. Furthermore, the sketch is much the same as the original sketch.
PG3_24_1
I wanted to use the sketch from the previous version to make a colour version of it. At first I used the colours of Joshua’s sketch.
PG3_24_2
In this version I have used the full colour spectrum. I find it a disadvantage that circles are depicted in the beginning that have no further relation with the rest of the animation. Maybe I can fix this 'bug' at a later date. Probably it also has to do with the fact that Joshua's audio file starts with a hard blow. My audio-file starts gradually.
PG3_25_1
I made a png file of six horizontal lines with rounded ends. They are at an angle of forty-five degrees. Furthermore, I have limited the color scheme. At least I added more gray values.
The animated version: https://vimeo.com/270627439
PG3_26_1
A sketch with svg files. In fact, I used the same cube three times. But they all react differently, which makes the image look different. The animated version: https://vimeo.com/270627840
PG3_27_1
This version uses HGridLayout. I also used the svg files from the previous sketch.
PG3_28_1
These Hexagons are positioned in a grid. Only their size is influenced by the audio-file. The animated version: https://vimeo.com/270628524
PG3_28_2
I increased the gridsize again. Decreased the maximum size of the cubes.
I have had enough of my audio-file. Listened to it much too often so I decided to make a new version which I will use for the rest of this course. Here it is: https://soundcloud.com/henklamers/pg3-8-2
PG3_28_3
In this sketch I have reduced the audio range to eight. That worked much better with this new audio file.
PG3_29_1
This is actually a bit too intense. But I'll leave it anyway. Until now, I have not really misbehaved.
PG3_30_1
I have removed the svg files. Instead there are very small cubes that are stretched in the z-direction. I still think the animation behaves quite nervous.
PG3_31_1
The cubes made twice as long in the z-axis. I also changed the lighting. Initially made its own lighting setting and removed the colors from the cubes. But if you leave the colors and use Joshua's lighting, you get a much better result.
PG_32_1
I have doubled the amount of cubes. In addition, I had a lot of work to position the geometry. I found the amount of cubes difficult to position. So I checked it by putting auto-clear on false. That gave me the following variations. This one is with rotateY.
PG_32_2
A version with rotateX.
PG_32_3
A version with rotateZ. Which incidentally produces a flat version in 3D.
PG_32_4
A combined version of rotateX, rotateY and rotateZ.
PG_33_1
AutoClear is on true again. It is almost right. When I look at this animation until the end, I see that the second part of the animation reacts better to the audio file. So I'm going to adjust the audio-file once more.
PG_33_2
This is a huge improvement. The animation is now much better in line with the sound. The quiet parts in the audio are also translated into little movements. So maybe it's wise to work on the animation in two ways. Create an audio file. Watch how the animation reacts to that. And adjust the audio-file so many times until it works fine with the animation. https://vimeo.com/272081601
PG_34_1
I do not know why there is now an extra large outline cube around all the cubes in Joshua’s animation. But perhaps it is interesting to make a variation with that.
PG_34_2
A version with outline cubes only.
PG3_35_1
In this version the large cube with its content is rotated based on sounds which are coming from the bass and snare in the audiofile. https://vimeo.com/272081751
PG3_36_1
I tried to use the cubes from the previous sketch for this new sketch. But that did not work out well. So I kept the spheres. What strikes me is that the HEllipse still gives strokes while noStroke is called.
PG3_37_1
In this sketch the smaller ellipses are used to form the light for the illumination of the large sphere. I have the idea that it is not quite optimal.
PG3_38_1
This has become an interesting picture. But I’ve got unreliable feedback from the audio file, It does not run one on one. But on the other hand I think it is pretty refined when a few objects don’t participate in the movement.
PG3_39_1
A small difference with the previous sketch is that variables have now been used for the speed in the Z and Y directions. There is also a minimum and a maximum limit. https://vimeo.com/272081914
PG3_40_1
I have reduced the amount of objects from 128 to 6. The large sphere is now also rotated. Although I am not sure about that. So I drew a cube around it and it just stood still. So the sphere does not rotate. It is an illusion.
PG3_41_1
I find the way in which you program a texture mapping on a sphere enormously cumbersome. I studied it, but I found it very complex. Especially if you compare it with the human-friendly way in which HYPE deals with code. So I left the more complex code for what he was and changed the texture file. That was pretty easy. Furthermore, I have increased the sphere in the center and the number of smaller spheres.
PG3_42_1
And this is the last exercise I did for this course. I repeated the center sphere again. And you are located in the second sphere. In that second sphere, the central sphere is hanging around with all the other smaller objects. The outer sphere is semi-transparent. https://vimeo.com/272081940
And if you wonder why the animation is so messy. And why it does not run synchronously with the music. That is because the sound of my keyboard is picked up by the microphone and that is converted into the movement of the animation. Spotify also helps a bit. But anyway... this is the end of this very instructive course Programming 3, Painting with sound. Thanks Joshua Davis!
