New School Year, still working on the same projects

Though do not let that sound resentful; it’s not at all.  I’m still glad to be working at Seneca’s Centre for Development of Open Technology.  It’s great to be able to do work on open source tech (and get paid for it).  Processing.js is coming to a close for several of us.  At least, with the release of 1.0, maintaining the library will become less of a daily thing for most of us.  There’s definitely still bugs to work on and if anyone out there is interested in helping out, just go to this link: here.  Sign up and take a bug to fix!

In the meantime, I’m continuing my work (that I’ve previously stalled) on XB Pointstream.  I’ll initially be working on a reader that converts a proprietary file format to something that’s more usable in the open web (and for our library).  This will probably take me a couple of weeks to get done.  After that, I’ll be working on the many bugs we have filed plus some that are still processing in our heads.  This is definitely going to be a fun semester!

0.9.7 Contribution

Well, I was kind of distraught in my last few posts.  I didn’t know what I was doing differently from some of the things other people were doing.  The code was almost exactly the same (minus obviously different variable names).  I found out later, after taking apart Processing.js and reducing it to less than 4000 lines (it’s currently sitting at 11000+ lines), that the problem was not the way I coded the 3D texture.  The problem was the way we were redrawing 3D objects without a call to the background function.  Or at least, if we didn’t call background in the redraw loop.  I ended up changing my examples to working ones by adding background to the redraw loop.

After I did that, I made sure to make a ticket to fix the background redraw problem.  Andor said he’d fix it, so I assigned it to him.  It made it to the repository easily enough as the fix was one line but I’ve yet to test it with my work.  I have been working on other things to make it for the 0.9.7 release.  Hopefully, I can test out the examples soon though.

The other function I was working on was textMode.  One particular function of textMode, the default, was already implemented.  The other use of the function was something that I didn’t fully understand and needed the use of the beginRaw function that we will not put in until later.  This textMode function was the reason I was looking at createGraphics again.  The functionality I was working on was the textMode(SCREEN) parameter.  This is basically a heads-up display (HUD).  It writes the text you want on screen and removes / disallows any transformation changes affecting objects in the scene.  So, it becomes written to the screen and unchangeable through transformation like a heads-up display in video games.

To get this working, I used another Processing instance (which is what createGraphics was for) and rendered the text there.  I, then, textured that Processing instance/canvas onto a quad according to the coordinates given.  I made sure that the quad wasn’t affected by transformation calls through pushMatrix and popMatrix functions.  I did find a problem when using a white fill on a transparent background and filed a ticket for it.

Still lost (createGraphics)

Well, it looks like I’m still on my own for now.  I’m still currently in the dark on reasons as to why it’s not working.  My last post indicated most of the code and test cases I’ve done.  I haven’t changed the code very much but I have improved on more specific test cases to try to get to the root of the cause.  Much to my dismay, however, the new cases haven’t revealed much.

Since my test cases before proved that this worked without Processing.js, the next logical step would be to incorporate Pjs into the equation and see what happens.  My first test case (picture below) involves implementing Pjs on a canvas and then throwing that canvas as a texture on a 3D WebGL object; in this case it’s a cube/box.

No problems, so far.  My next test case switches roles for the two canvases.  It comprises of Pjs texturing a quad.  The texture comes from a native canvas with no Pjs on it.  Nothing broken, still.  The next test case was given to me by one of my CDOT co-workers, Matt Postill.  This test case entails that both canvases use Pjs.  However, instead of using createGraphics to implement a Processing.js instance through the original Pjs instance, both canvases already have Pjs and we’re connecting them through the texturing of the quad.

This still works.  At this point, I’m running out of ideas and I’m starting to grasp at straws.  I do still have an idea, though.  My theory, since the quad texture is picking up the supposed first frame of the second canvas, is that the texture is only seeing the original canvas state and not seeing any updates to the canvas through Pjs (this is actually a stupid theory in hindsight considering Pjs does all it’s drawing and filling through canvas functions).  I test this by drawing and filling a square in the sketch through the native canvas functions.  This is broken but I was hoping it would work in this instance.  And I’m still at the point I was at since my last post… lost.  My next step will probably be to strip Pjs of unnecessary components and try to see if it’s anything in Pjs that’s preventing the redraw of the little canvas in the big canvas.

Looking for help… (more createGraphics stuff)

I’m at a block here…

When I last talked about createGraphics, I said I had it working with 3D.  This was partly true.  I had an oversight and assumed all was done.  3D was working as long as the initial context for the sketch was 2D.  I figured this out last week as I was trying to use createGraphics for textMode.  My naïvety made me think this would just work while the context was originally 3D.  Of course, it didn’t.  Trying to render 2D or 3D on an original 3D context didn’t work.  That’s because we didn’t handle it.  Now, I’m trying to fix that to implement textMode.

Since I couldn’t get it to work, I started by stripping Processing.js from my test case.  I went with Learning WebGL lesson 5.  This was my initial test case and my objective at the beginning was to use another canvas as the texture.  That was simple enough to get working and the example of that can be seen below.

So after I got that working, I did a double objective test.  The first objective was to get the test working with a dynamically created canvas (like I did before).  The second objective was to try and make it interactive (kind of).  The interactivity, well isn’t really interactive.  I just basically got the background of the canvas to change colours.  I, also, compared code to another example much like the one I’m trying to create.  Anyway, here’s my example:

This seemed to work okay as well, so my next task is putting it in Pjs.  This is the portion I need advice/direction on.  I applied all my knowledge of the existing subject and am still having trouble.  The test case is a little different as I’m rendering a quad instead of a box and the actual object is more interactive than just the background of the canvas changing colours.  However, it should still work and for some reason the image inside isn’t refreshing.  Here’s the test case online:

Mouse over the middle of the left canvas

Now that I’ve given you background on what my objective is, I’ll start talking about the code.  First, we’ll begin with the Pjs sketch.  I’ve stripped out and commented the unnecessary code.  That’s why the background for the bigger canvas is defaulted to grey.  So, the only really working factor is the quad is getting drawn and it’s using the smaller canvas as it’s origin.  The important part of the sketch is the image() function… it’s the point where the smaller canvas is being rendered onto the quad.

I took the important part of the image function out and put it in this pastebin.  This is the part that applies to this subject.  Basically, it takes all the information from the p.texture and p.vertex calls and renders the quad at p.endShape.  p.texture is where the texture is binded to the quad.  Here’s the important part.  Now, the binding for this almost exactly the same as my Learning WebGL example above.  However, I’m still getting problems with the redraw/re-render.  So, it could be within the endShape that I’m getting problems.  The section that renders the quad is called fill3D.  That’s where the points and values get passed onto the program object which uses our shaders.  Now, if you’re wondering about the shaders… here they are.  Could anyone out there guide me on where I’m going wrong?  I am stumped.

Key stress

For the last week and a half, I’ve been working on making Processing.js (Pjs) keys and keystrokes work more like its Processing (p5) counterpart.  Actually, it was originally already set for review but the reviewers didn’t wish to implement browser detection.  So, I’ve been focusing on feature detection for keys within Pjs.  Now this fix doesn’t seem very hard, only if the browser vendors actually standardized their key presses and key strokes.  Instead, they like to use the method of “ours is better, so do it our way”, which isn’t very helpful to the developers.  I’ll explain my situation and why I’m so stressed and frustrated about this particular fix.

Key codes are not a particular problem within this situation.  Most of the keys/code values are either similar or there’s already existing code to account for different codes.  Not something I’m worried about in terms of Pjs.  Key strokes and repeating is the bigger hassle in this.  There’s also the fact that some browsers use charCode instead of keyCode or vice versa.  Those two combinations by themselves can create a huge mess.  I’m just glad the actual codes aren’t completely different as well, or this would be an even bigger headache.

I’ll start off by saying that there’s three different type of key press events. First is keydown, when you push down a key. Then it’s counterpart is keyup, when you release a key. The last is keypress, which determines if a key is being held down. Coded keys are keys in the keyboard that can alter the given keycode when a key is pressed. The five main keys are Ctrl, Alt, Shift, Caps Lock and Num Lock. Char keys are essentially anything that can produce a character. The movement keys allow movement through the document and then there’s the F keys.

Difference in Key Strokes and Key Uses

Processing	Firefox	Chrome
char keys refires all events when key is being held down	char keys refires keypress event when key is being held down	char keys refires keypress and keydown event when key is being held down
coded keys do not get refired when being held down, keypress value does not exist	coded keys do not get refired when being held down, keypress value does not exist	coded keys do not get refired when being held down, keypress value does not exist
movement and f keys refire keydown and keyup events when held down, keypress value does not exist	movement and f keys refire the keypress event when held down	movement and fkeys refire the keydown event when held down, keypress value does not exist
All platforms deploy keydown event when a key is pressed and keyup event when a key is released.

They may seem like minor differences but setting them from two differing places to one uniform place creates a lot of problems.  When I was correcting small bugs that came up, another one would pop up and it just kept going until I lost track of what fixes I did for certain bugs.  I’d end up having to restart from a clean slate to begin the comprehension again.  Anyway, I know people are looking to get this working properly as I’m getting hits on my blog about keys in Pjs.  I’ll make sure it gets in before 1.0!

Toronto WebGL Community

Left to Right: myself, Benoit, Andor and Matt

This is the Toronto WebGL Community!  Or at least the people we know that currently actively work on it…  if there’s more of you out there, feel free to let yourself known and get involved by posting on the comments of this blog.  I’ll make sure the right people see your stuff.

Anyway, this picture was taken at the Mozilla Toronto Offices.  Yesterday, I spent the afternoon with some of my co-workers talking to Benoit Jacob about the WebGL implementation within Minefield.  We were doing some profiling and giving feedback on what changes could be made to provide better support for those using the WebGL API through Firefox/Minefield.  So if any of you have suggestions, specifically more about how it’s implemented in Minefield and less about the actual WebGL standard, feel free to leave a comment below and I’ll do my best to get the idea to them.

I, also, learned a couple of neat tricks.  One such is a built in profiler in Linux that I could use on my desktop at work.  It will make bottlenecks a lot easier to find.  The command is perf record, which actually only came out to Ubuntu on a recent release.

We discussed other performance boosters, like TypedArrays.  This is a new proposal to the Javascript language and will be somewhat monumental considering JS is a type-less language.  This introduction will help improve WebGL code and will pave the way for the final release of WebGL.  It just has to be approved by the standards committee.

***NEW*** Toronto WebGL Mailing List (created by Benoit) – clickity

createGraphics 3D (PGraphics) working!

I actually got it working last Tuesday and the working example of it has been up since then.  I just wanted to polish it more and maybe make a different example to show the process in this writeup.  However, I’ve been busy since last Tuesday and prolonging this announcement seems unwise so I’m just going to go with what I have.

createGraphics now runs with a 3D environment!

Note: you need a webGL enabled browser to see 3D

After a couple of months of thinking and deciphering the brilliant hack that John Resig put in for createGraphics/PGraphics, I finally got its 3D counterpart working.  Actually, it wasn’t really a couple of months… maybe altogether about 2 straight weeks.  I’ve had to test whether it was possible first and the results were very good.  It also helped me file bug 571061 with Mozilla.  Once I had it working without Processing.js (PJS), I knew what the process was to get it working with PJS.  At first, I implemented it with pre-existing canvases.  Once that worked, I changed it to work with dynamically created ones.  It turns out that I was already really close and in the end the code only needed a few modifications.  And now we have createGraphics working with the 3D scope as well!  Maybe I’ll put up a better example sometime soon involving great uses for this new feature, like mirrors suggested by F1LT3R.

Curved lines in 3D context!

0.9.3 code freeze has come and gone.  Well, it’s not really a code freeze; more of the fact that the main PJS development team will be looking at other projects and sub-projects for a couple of weeks in the summer.  So, the development of PJS will be a little slower over the next month or so.  For 0.9.3, we pushed out as much working code as possible and moved the rest that we deemed unimportant enough before this freeze.  There are still a few major functions that we have to get to and probably will be taken care of before 1.0 is released.  Anyway, I just wanted to give you this brief update but what I really wanted to talk about are the bezier and curve functions.

These primitives are mainly used in 2D sketches for PJS; only because 3D didn’t support them yet.  Now, we can get sketches like the following working in browsers:

(Note: WebGL enabled Browser needed)

That demo is a little slow and needs major optimizations on our part.  Also, there are still known bugs that we are looking to get fixed at a later date.  Trying to fill curves or beziers right now does not work very well, so I would suggest not doing it (if anyone can provide information on this subject, it’ll make for a quicker fix).  Despite fills not working in curved lines, I was able to get it working (using a workaround) on 3D ellipses.  However, it does come with it’s own flaw where in pixels are fighting for space.  I, already, have a working idea on how to fix that.  It involves dynamically setting the polygon offset on each new created object to be 1 pixel higher than the previous one in those coordinates.  It sounds simple to explain but I have a feeling I will run into some expected and unexpected problems.  We’ll see.  Before I get to these new bugs though, I’m going to fix PGraphics slowly.

PGraphics cont’d…

I said I was finished with working on PGraphics; turns out I’m a liar.  I was working on it for a day more, with much encouragement from Dave.  Through our meeting yesterday, he helped me see the model of disassembly to see if it was possible to even do.  For debugging the issue, he suggested I strip PJS from the equation and see what we can do with just canvas.  So, I took the source code for one of the Learning WebGL examples and used it for my test case.  I was having more problems with it and was almost ready to give up.  Dave suggested that I strip down the example more; remove the movement from the objects and pre-render the initial canvases instead of writing them in the background.  In the end, I had a working test case.

(Note: need a WebGL enabled browser)

The problem I was starting to experience at the end of working last night was that when writing to a buffer (instead of a pre-rendered canvas), I was getting errors on Minefield.  The new example was working on Webkit/Safari though (with a few resize bugs).  I think the bugs/problem had some sort of size issue with the buffer but I couldn’t look into it more.  I’m moving past this problem for now as I have other items I need to fix (and a greater amount of them).  Hopefully, with this basis, it won’t be too hard to solve the problem the next time I look at it.

Drawing a 3D Canvas on top of a 2D Canvas

… is apparently not yet possible on Minefield (as per my conversation on Mozilla IRC).  It’s not possible in other browsers yet either.

<aSydiK`work> does anyone know when trying to change data in an ImageData object, if the data values have to be exactly the same?
<aSydiK`work> I’m reading data that comes out as Uint8Array and need to put it into a Uint8ClampedArray and it’s not going through
<bz> hmm?
<bz> how so?
<aSydiK`work> I’m using canvas.context.createImageData and I’m trying to change the data in there but it’s of type Uint8ClampedArray and I’m trying to throw in a type Uint8Array
<aSydiK`work> I was just wondering if it was even possible
<vlad> aSydiK`work: why would you want do that?
<vlad> the ImageData object requires the semantics of Uint8ClampedArray
<aSydiK`work> well basically, I’m trying to get a webgl-context canvas into a 2d-context canvas
<vlad> to draw from one to the other?
<vlad> in theory drawImage will take any canvas, but that’s broken currently for webgl canvas sources
<aSydiK`work> I’m trying to draw the 3d canvas onto the 2d one
<vlad> I could just fix that for you 🙂
<aSydiK`work> that would be awesome… it’s for processing.js 😀
<aSydiK`work> vlad: what should I do to expedite the process?  did you need me to file a bug or something?
<vlad> there is one already I believe
<aSydiK`work> awesome

After spending the day trying to get a workaround with the image part of the library, I ran into a roadblock.  Unfortunately, this means that the 3D functionality of PGraphics will have to be delayed until canvas.context.drawImage is implemented within the 3d scope.  I’ve been trying to imitate what the already existing 2D PGraphics does in PJS.  2D PGraphics was written to take a new canvas, plaster pixel data on it and then draw the canvas on top of an existing canvas.  I tried doing the same thing through WebGL’s readPixel function.  readPixel returns a Uint8Array data object.  I needed to use an ImageData format… so I built a 2D canvas, called createImageData and tried to set the data.  My final outcome was the following (note – you need a WebGL enabled browser to see):

This is where I was running into formatting issues.  It just wouldn’t render the middle canvas.  So I asked for help, which led me to the #jsapi channel in Mozilla IRC.  And that’s where Vlad told me they were working on it.  Hopefully, it gets in soon so I can work put PGraphics in PJS!