createGraphics 3D (PGraphics) working!

I actually got it working last Tuesday and the working example of it has been up since then.  I just wanted to polish it more and maybe make a different example to show the process in this writeup.  However, I’ve been busy since last Tuesday and prolonging this announcement seems unwise so I’m just going to go with what I have.

createGraphics now runs with a 3D environment!

Note: you need a webGL enabled browser to see 3D

After a couple of months of thinking and deciphering the brilliant hack that John Resig put in for createGraphics/PGraphics, I finally got its 3D counterpart working.  Actually, it wasn’t really a couple of months… maybe altogether about 2 straight weeks.  I’ve had to test whether it was possible first and the results were very good.  It also helped me file bug 571061 with Mozilla.  Once I had it working without Processing.js (PJS), I knew what the process was to get it working with PJS.  At first, I implemented it with pre-existing canvases.  Once that worked, I changed it to work with dynamically created ones.  It turns out that I was already really close and in the end the code only needed a few modifications.  And now we have createGraphics working with the 3D scope as well!  Maybe I’ll put up a better example sometime soon involving great uses for this new feature, like mirrors suggested by F1LT3R.

Manually creating a PNG…

… pixel by pixel.

Like I’ve said before, most of the Processing.js team are shifting their main focus on different projects over the summer.  Some of the members are working on Popcorn.js, a Javascript video player library that uses the HTML5 video tag.  Andor Salga and I are working on imaging 3D point cloud data for Arius3D.  Andor already has a working prototype gallery for what we have to do with our point cloud data; his blog about it is located here.

I have been working on the non-3D spectrum of this work and trying to improve the load times for our prototype.  Well, that’s what I’m going to try and accomplish in the end anyway.  I’ve been looking for ways to solve this problem and the first method I decided to try was server side scripting using PHP.  While PHP isn’t a hard language to comprehend, I soon ran into problems concerning the use of the data I was reading from file.  The only reason I was using PHP was to try and read the file data to give to Javascript.  If trying to extract the data was this much of a hassle, there was no point in using it.

I moved on to just reading the file straight from text using XMLHttpRequest.  I had problems with putting setInterval and setTimeout methods within the code to try and run asynchronously.  The problem was that I didn’t need them!  After I removed them and just called the function on its own, I got the stream and conversion to work… but very slowly.  An example of the stream can be viewed below… be warned, though, it does take a number of seconds.

I was talking with notmasteryet, one of the audio wizards that came up with this hack of streaming byte data into a png for Text to Speech use within browsers, and he told me to use bit shifting instead of string conversions to plug the information into the png.  This would of course make the conversion MUCH faster.  However, something is wrong and not working as it should.  I’m writing this blog not due to accomplishment but more of a bookmark on what I may have to continue with later.  I’m moving on to dissect the PSI Reader given to us by Arius3D and trying to convert it to Javascript to be used for on the fly conversions.

Curved lines in 3D context!

0.9.3 code freeze has come and gone.  Well, it’s not really a code freeze; more of the fact that the main PJS development team will be looking at other projects and sub-projects for a couple of weeks in the summer.  So, the development of PJS will be a little slower over the next month or so.  For 0.9.3, we pushed out as much working code as possible and moved the rest that we deemed unimportant enough before this freeze.  There are still a few major functions that we have to get to and probably will be taken care of before 1.0 is released.  Anyway, I just wanted to give you this brief update but what I really wanted to talk about are the bezier and curve functions.

These primitives are mainly used in 2D sketches for PJS; only because 3D didn’t support them yet.  Now, we can get sketches like the following working in browsers:

(Note: WebGL enabled Browser needed)

That demo is a little slow and needs major optimizations on our part.  Also, there are still known bugs that we are looking to get fixed at a later date.  Trying to fill curves or beziers right now does not work very well, so I would suggest not doing it (if anyone can provide information on this subject, it’ll make for a quicker fix).  Despite fills not working in curved lines, I was able to get it working (using a workaround) on 3D ellipses.  However, it does come with it’s own flaw where in pixels are fighting for space.  I, already, have a working idea on how to fix that.  It involves dynamically setting the polygon offset on each new created object to be 1 pixel higher than the previous one in those coordinates.  It sounds simple to explain but I have a feeling I will run into some expected and unexpected problems.  We’ll see.  Before I get to these new bugs though, I’m going to fix PGraphics slowly.

PGraphics cont’d…

I said I was finished with working on PGraphics; turns out I’m a liar.  I was working on it for a day more, with much encouragement from Dave.  Through our meeting yesterday, he helped me see the model of disassembly to see if it was possible to even do.  For debugging the issue, he suggested I strip PJS from the equation and see what we can do with just canvas.  So, I took the source code for one of the Learning WebGL examples and used it for my test case.  I was having more problems with it and was almost ready to give up.  Dave suggested that I strip down the example more; remove the movement from the objects and pre-render the initial canvases instead of writing them in the background.  In the end, I had a working test case.

(Note: need a WebGL enabled browser)

The problem I was starting to experience at the end of working last night was that when writing to a buffer (instead of a pre-rendered canvas), I was getting errors on Minefield.  The new example was working on Webkit/Safari though (with a few resize bugs).  I think the bugs/problem had some sort of size issue with the buffer but I couldn’t look into it more.  I’m moving past this problem for now as I have other items I need to fix (and a greater amount of them).  Hopefully, with this basis, it won’t be too hard to solve the problem the next time I look at it.

Drawing a 3D Canvas on top of a 2D Canvas

… is apparently not yet possible on Minefield (as per my conversation on Mozilla IRC).  It’s not possible in other browsers yet either.

<aSydiK`work> does anyone know when trying to change data in an ImageData object, if the data values have to be exactly the same?
<aSydiK`work> I’m reading data that comes out as Uint8Array and need to put it into a Uint8ClampedArray and it’s not going through
<bz> hmm?
<bz> how so?
<aSydiK`work> I’m using canvas.context.createImageData and I’m trying to change the data in there but it’s of type Uint8ClampedArray and I’m trying to throw in a type Uint8Array
<aSydiK`work> I was just wondering if it was even possible
<vlad> aSydiK`work: why would you want do that?
<vlad> the ImageData object requires the semantics of Uint8ClampedArray
<aSydiK`work> well basically, I’m trying to get a webgl-context canvas into a 2d-context canvas
<vlad> to draw from one to the other?
<vlad> in theory drawImage will take any canvas, but that’s broken currently for webgl canvas sources
<aSydiK`work> I’m trying to draw the 3d canvas onto the 2d one
<vlad> I could just fix that for you 🙂
<aSydiK`work> that would be awesome… it’s for processing.js 😀
<aSydiK`work> vlad: what should I do to expedite the process?  did you need me to file a bug or something?
<vlad> there is one already I believe
<aSydiK`work> awesome

After spending the day trying to get a workaround with the image part of the library, I ran into a roadblock.  Unfortunately, this means that the 3D functionality of PGraphics will have to be delayed until canvas.context.drawImage is implemented within the 3d scope.  I’ve been trying to imitate what the already existing 2D PGraphics does in PJS.  2D PGraphics was written to take a new canvas, plaster pixel data on it and then draw the canvas on top of an existing canvas.  I tried doing the same thing through WebGL’s readPixel function.  readPixel returns a Uint8Array data object.  I needed to use an ImageData format… so I built a 2D canvas, called createImageData and tried to set the data.  My final outcome was the following (note – you need a WebGL enabled browser to see):

This is where I was running into formatting issues.  It just wouldn’t render the middle canvas.  So I asked for help, which led me to the #jsapi channel in Mozilla IRC.  And that’s where Vlad told me they were working on it.  Hopefully, it gets in soon so I can work put PGraphics in PJS!

Processing.js and cross browser keycode compatibility

On Friday, I began looking (in more depth) at a ticket that started out as a simple problem of figuring out why an example sketch wasn’t working exactly like it’s Processing counterpart.  It started out as a simple problem but turned into the urge to overhaul our current key input structure to something that works on all browsers.  After much testing and figuring out how each browser treats keyCodes and key strokes, it is my recommendation to add a browser check and account for how each browser handles these items.

I used w3.org’s Keyboard Events and Codes page to figure out what each browser spit out as different keyCodes and charCodes.  This helped me figure out that Opera is pitching out the % keyCode for pressing the left arrow key.  As for key strokes, which was the original problem in the ticket, I used this site.  It showed me that Chrome refires keydown events (on certain keys) when you hold a key down.  This explains why the sketch works properly in Chrome but not in Firefox.  To make it work like p5, we’d have to account for all of that and eventually make everything uniform.

For implementation, I’m looking to port directly into PJS some browser and operating system detection code written in Javascript.  It uses the information from navigator.userAgent.  Strings may not be the best way to deal with them but at the moment it’s the only thing I’ve found that’s simple enough to implement.  If anyone’s got better ideas, feel free to link them to me here.  After I get the browser detection working, I’m moving on to making sure Firefox and Chrome work as p5 does.  Implementing the other browsers may be moved to a later date depending how much I can get done.

My only question to the browser builders out there is… couldn’t you guys get together and standardize all this so users don’t have to worry so much about browser compatibility?  *insert frowning face*

Building mozilla-central/Minefield on 64-bit Windows 7

Let me start off by saying that I didn’t actually start or finish this build.  Maybe I’ll do it sometime on my own time, but during work, it would be too much effort to try to do this build.  I was doing research to try and solve my problem already and I figure I would share what I found with the world.  There’s not much documentation out there for this build so I’m hoping this can eventually help someone else.

To begin, there’s no straight forward linear build for the Windows 7.  Well, the Simple Firefox Build on the Mozilla Developer Centre Wiki provides you with simple enough instructions but the builds for Linux and Mac OSX are way easier.  This is mostly due to Windows not being the ideal development platform.  Anyway following the steps provided in the build instructions, you would end up on the Windows SDK Versions page.  There’s a warning on this page for 64-bit users and it states:

Note that if you’re running a 64 bit version of Windows you may require the “AMD” version of the SDK even if you have an Intel processor, and you may need to deselect the documentation component in the installer in order to avoid an installation error. (This was certainly the case for 64 bit Windows 7 on a late 2008 MacBook Pro.)

That confused me a little, as I couldn’t find the AMD version of the SDK.  I tried installing the regular SDK and it installed without a hitch.  I figured that if there was a OS-bit problem that there would be an error thrown.  Then, the build started and finished.  No problems.  Later, testing the build showed me there were problems.  It didn’t build properly and feedback told me that the build was actually 32-bit.  So, here starts the 64-bit research and the reason I created this blog.

Googling my problem, one of the first links that catch my eye is in the mozilla.dev.build google group.  It’s about build on 64-bit Windows 7 SP2 started by Armen Zambranio Gasparnian, whom I actually recognized was from Seneca’s Open Source Development course (like myself).  First thought was Awesome! Reading on though, I realized he had trouble building it.  Not only that, but his platform is completely different from mine and far more cutting edge.  He’s working with Microsoft Server 2008 along with Visual Studio 2010.  I close the window, for now and move on.

Later, I’m relayed a message originating from Ted Mielczarek, a platform engineer from Mozilla Corp.  The message concerned going back to Armen’s blog for a second look.  I looked through better and found his older blog post.  While the post has mainly to do with the aforementioned platform (MS2008 and VS2010), a comment at the bottom caught my eye concerning WinXP 64-bit building.  The comment mentions involving having to install unofficial patches from this site, which I fail to mention that I’ve seen before through my googling.

Finding this information, I talk to Dave.  Through his info from Ted, along with what I found out, it was decided that the process would take too long and was not really needed (as the 32-bit version of Minefield worked just as well on a 64-bit Windows 7).  So the build was scrapped, but I’ll look into it again if I ever get some free time at work.

Long Journey

We’ve come a long way from when we first started the Processing.js project.  I came to this realization while looking for demos and cool projects to showcase at the OCE Discovery convention happening next week.  At first, I was having trouble finding something new because most of the cool stuff happening with PJS was something we already knew about (for example, Dave’s Audio Build with PJS).  So, trying to find something new was a bit of an arduous task.  When I had given up and looking on to do better things, two different uses/visualizations show up over the weekend.

The first is a visualization of the Evolution of Privacy on Facebook.  This was made by Matt McKeon who is a developer of the Visual Communication Lab at IBM Research’s Center for Social Software.  While the demo isn’t as flashy as some of the stuff that myself and the PJS team have seen before, it’s really impressive as far as seeing the uses for the library.  Speaking for myself, it caught me by surprise.  I started realizing that PJS was becoming more mainstream.

My second example came over the weekend in the form of a game.  Being a gamer myself, I was pretty excited to see Zelda on PJS.  This was created by Derek Doucett at ravenousgames.com.  It’s still a little slow and the PJS code side of things probably needs a lot more optimization but it’s definitely really cool.  Things like this help with determining what we need to work on with optimizing the library code.  It’s really great to see our work being used for amazing stuff like this.

Unfinished Business 0.8 – 0.9 – 1.0

This post is looooonnnnngggg overdue, considering my last post was in March.  I figured I could do this while Minefield was building (but I was wrong considering it only takes like 20 mins for it to build on these machines).

0.8 Release

I really didn’t do all too much for the 0.8 release.  I was busy with other things.  I realize, now, that I should’ve thought more importantly of this project and should’ve made as much time as possible for it.  Unfortunately, there’s not much I can do about the past but learn from it and dedicate my time better.  I did feel badly for not doing as much for the 0.8 release as my colleagues were depending on me and hopefully made up for it somewhat with my work with the 0.9 release.

0.9 Release

For 0.9, I worked on beginShape, vertex and endShape.  I had a barely working product for 3D (untested and unused) before the Processing.js workshop.  It was barely working due to the fact that my changes broke the 2D version.  During the workshop, I was able to run tests and completely revamp the way beginShape/endShape worked.  With the 2D version, the functions were set to draw as the vertices were put in.  I changed it so that it collects that information and then renders at the end.  I, also, spent a whole day on fixing bugs and testing the code.  This was needed due to the fact that it was a large amount of code.  My examples and tests can be found on at the following link:  click here.  (Note: that 3D tests and demos require a pre-release browser that supports WebGL… such as Minefield, Chromium or Webkit).  Here’s a more prominent 3D demo using beginShape/endShape (again you need a 3D capable browser):

After that weekend, I was able to work on converting 2D quad and triangle primitives to work in 3D space.  This wasn’t really too tough as it was merely a wrapper

1.0 and Future Releases

1.0 is a goal for the end of May… however, we will be doing mini-releases along the way to sort out some fixes and release some more needed features.  My first goal is to make that RGB Cube demo working up to speed.  This entails converting a bit of the current endShape code to take in colours for the vertices and render them.  I, also, have to make sure that these objects take normals so they can be lit up in a 3D environment.  After, I plan to work on the PGraphics object and make sure that the fix works with 3D objects being displayed in createGraphics.  My final plan of action before the 1.0 release date is to ensure the PShape datatype is working within the PJS environment.  Of course, I’ll also be fixing any bugs that come along the way.

0.7 release

I was a little hard-pressed for this week’s release.  Mainly because I thought the release to be next week.  I did have some stuff done before but I wasn’t in much of a rush to finish it, until I found out it was to be out this Monday morning.  I got in gear but still had much trouble with beginCamera() and endCamera().  The functions you see there seem pretty simple but having to use the inverse of camera threw me for a loop.  I had to look for the problem everywhere in the code.  At one point, I rewrote the whole invert function only to find out it was working properly.  Some key objects I was missing were dependent on deprecated functions.  At least, the code within the Processing native said they were to be removed from the future.  In that sense, my colleague decided it wasn’t needed in PJS.  I quite agree with him still.  It’s not completely needed, some of those inverse functions anyway.  I could and will probably recycle some of the regular functions and just apply the inverse.  I did add them all for now though.

That’s not all there is to it either.  As I was submitting an example through git, I found a bug that still currently resides with my checked-in code.  It may seem like it works properly, but insert a camera() function in there and the changes don’t take.  That is definitely a problem because suppose it’s put in a draw{} statement.  It wouldn’t have an anchor that can put it back to place… it’ll just continue to transform.  It’s somewhat of a big bug that I can’t finish tonight.  I’m leaving it up to the reviewers to pass or fail.  I doubt it’s going to pass… I’m not sure I would pass it either.

On a lighter note, I was able to finish the dist-3d days ago.  I was just so wrapped up with beginCamera() and endCamera() that I never wrote the tests for it.  I wrote my first unit tests earlier today and it was a breeze.  I’m definitely glad that was added to the PJS scope.  It definitely makes unit tests much easier.

Anyway, off to bed.  Maybe someone will have read this blog and figured out a way to help me with the camera.  Dave was right.  It was my fault for waiting til the last second to ask for help.  I’ll blog more soon… still have to get PGraphics off the ground.