Haiku Blog-O-Sphere

Since my last blog entry a lot of progress has been made. Currently I'm right on the cusp of actually producing images on the screen that have been captured by my camera. Successful communication is occurring between the driver and the camera in at least two different forms.

The first form of communication to be successfully implemented involves the setting of values within the camera which affect image capture. These are the familiar brightness, contrast, sharpness etc. settings which most cameras support. Nearly all of the options available to my camera are now presented for manipulation by end-users and successfully communicated to the camera. These values are maintained within the camera between power cycles and this fact is successfully communicated to the user via the available controls. The controls can be viewed and modified in the media preferences application or the cortex demo application. ParameterWeb control documentation indicates a range of different style controls within the continuous or discrete parameter varieties. However it appears to me that the only discrete value input method currently supported with an appropriate gui interface is the binary on/off option. This is suitable for features like the auto value for white balance which can only be either on or off. However the powerline frequency setting which has three possible values was unable to be represented with the appropriate discrete control of the B_RECSTATE type which has three possible states. To simulate this capability a continuous control was modified to only allow three values which are indicated by placing the sliding control within a +/-10 range of the desired value. The slider snaps to the available values to indicate this behaviour. One future feature which would perhaps be desirable is controls with auto settings which indicate in real time by their movement what values the camera is using in its auto mode. Right now sliders are simply frozen in their last position when the auto mode is in effect. I had some brief discussion with my mentors about this feature, but it was deemed to be unnecessary at this stage as a lot of work is left to be done in actual image capture.

read more

The following objects have at least partial bindings:

From the Application Kit: From the Interface Kit: From the Support Kit: Application
Clipboard
Cursor
Handler
Invoker
Looper
Message Alert
Box
Button
Control
Font
Menu
MenuBar
MenuField
MenuItem
Point
PopUpMenu
Rect
Screen
StringView
TextControl
TextView
View
Window Archivable

At the moment, some of the classes are not very usable; they're necessary because other objects inherit from them. They will later be expanded to allow creation of custom objects based on them, so users could subclass a Looper or a View the same way they can now subclass Application and Window.

Python has a minimal test program that uses Application, Window, and Button, but the other objects have not been tested. It would be helpful if interested Python programmers could start coding (and thereby discovering bugs). In addition, I get a large number of warnings about multi-character constants when compiling. Despite following the instructions in the documentation, I cannot get the installer to pass options to the compiler to turn these off. There are also a few other warnings I'm working on eliminating, but the extensions successfully compile and the test program works.

For the time being, the Python objects are all named Haiku.Object instead of Haiku.Kit.Object. This allows me to split up the kits into different extensions; since an extension does not know what kit a foreign object is defined in, the kit name cannot be part of the object name. Since this naming scheme is apparently not good Python practice, I am still looking into alternatives; currently I am considering either placing everything into a single extension or adding some data to my definition files that lets an extension know what kit/extension a foreign object belongs to.

Perl has the minimal test program and it also has a slightly more complex test program (a small Person viewer app). There are also a few compiler warnings I'm still working on eliminating.

The bindings can be found at http://dev.osdrawer.net/projects/perl-haiku-kits. The downloadable files are under "Files"; there are also Forums and a Wiki, and (under "Issues") a bug tracker. Anyone can download the files and look at the content, but if you want to report bugs, post to the forums, or edit the wiki, you need to be added to the project. Interested users with an OsDrawer.net account can email me or post a comment here, and I will add you as a member of the project.

There may be some trial and error involved in adding members; OsDrawer.net has defined some roles for members of a project, but I can't find any documentation on what permissions each role has. There is a role called "Reporter" that I assume lets the user post new bugs, and there's one called "Wiki editor" that's self-explanatory. But there's no role for Forum poster, so I assume one or more of the other roles has that permission included. But I don't know which one(s).

Okay, let's take a look at my second-quarter goals:

- Bring the Python bindings to minimal functionality (Done)
- Write a minimal Python test program (Done)
- Continue to test threading issues (See below)
- Expand preliminary bindings and add new bindings (Done)
- Write test programs for the bindings (Partially done)
- Write documentation for the bindings (See below)
- If there is sufficient time, select a third target language (Insufficient time)

Threading: I have found several issues I thought might be threading issues, but upon closer examination, they were not. I did find one genuine threading issue - but it was because I forgot to lock a window before updating data.

Documentation: I have written some documentation and I am working on programmatically adding it to the bindings. Python has fields in the underlying C++ structures to add documentation, and Perl allows documentation to be mixed in with the code.

Now for my upcoming goals. In general, during the next quarter I will continue to keep an eye out for threading issues, add new bindings, and fix reported bugs. There are also a few more specific things I want to work on:

There are a number of methods that haven't been implemented yet because they have structs as input or output, and the bindings do not handle structs yet. It should not be too difficult to map these to appropriate data types in the target language (Perl hash, Python dictionary).

I'm not sure what to do about globals like be_app and be_clipboard; on the one hand, they could be treated like constants and restricted to a particular namespace, in order to not pollute the global namespace. This is the way I'm leaning right now. On the other hand, there are relatively few of them, and so it would probably not result in a great deal of pollution if I were to put them in the global namespace.

Several of the C++ objects have overloaded operators. I would like to expand the bindings to support these overloaded operators. I'm not sure how much time I want to spend on this issue right now, though. It depends on how much users want this feature.

- Continue to test threading issues
- Expand existing bindings and add new bindings
- Fix bugs reported by users
- Enable structs
- Expose globals
- Add documentation
- Write additional and more complex test programs
- If there is sufficient time, select a third target language
- If there is sufficient time and user interest, work on overloaded operators

Here is a topic that for the longest time as a BeOS/Haiku developer, I had no idea how to do and didn't really feel enthused on the concept of learning. Of course, now all the major platforms have desktop gadgets. Ironically, writing a replicant is really easy if you already know how to write basic Haiku GUI applications already. Find out how in this lesson.

Programming with Haiku, Lesson 21

Time does fly. It has already been two weeks since my previous blog post, so I guess an update is in order. The support for shine-through directories and the package links has been implemented as planned and works nicely as far as I have tested them. Other than that I've worked on a few odds and ends.

read more

I've just uploaded some files onto http://dev.osdrawer.net/projects/perl-haiku-kits/files They are minimal implementations (window with a button) for Perl and Python. The Perl one works fine, but I'm still having issues with the Python.

Here are some of the issues with Python:

Apparently by convention packages start with a lower-case letter. The bindings are currently in the 'Haiku' package. This would be a trivial change; it depends on how important it is to Haiku's Python user community.

PyModules and PyTypes (= classes) cannot share the same name, as far as I can tell. Thus, in order to have constants and plain functions available from (for example) Haiku.Application, the Application PyType must be in a separate namespace, currently Haiku.Application.Application. Depending on what the community wants, I could move all the constants and plain functions into the ApplicationKit PyModule, and have the PyType be Haiku.Application.

The really big problem, however, is passing objects. BWindow::MessageReceived gets a BMessage object. In order to pass this to Python, I need to have a PyType. But the PyType in question is defined in the ApplicationKit, which is a separate package. So I don't have access to it.

In Perl, this was not a problem; I simply used a string containing the Perl class name, and as long as the user had loaded the relevant package, Perl took care of it. But Python wants the PyType, not just a string. I'm still looking into a way to get around this problem. I'm trying to do it by eval'ing some Python code, but so far I have been unsuccessful. If I can't do it any other way, I could export the PyType from the other package, but then Python would be loading the .so and I would be loading it a second time, in addition to the extra overhead of exporting and importing. It just seems like a waste of resources.

In any case, the Python test script displays the window, and calls event hooks on the Application object (ArgvReceived, ReadyToRun, QuitRequested), but when you click the button and it tries to call the Window's MessageReceived event, it dies.

A lot of commits have happened since I last blogged on my GSOC Project. The big hitters are:

• the framework of a rankings website for users’ point totals
• A non-regex parser I wrote because I’m impatient (I’m going to test its speed.)
• The begining of the http server that will inform user’s builddrones of availible jobs (builds) [Basically it tells you if packages have been updated so you can build them]
• Verified that Camlistore (on python) can work on Haiku.
• A Jenkins API wrapper for magic to happen (I think it may be the perfect thing to handle build logs and results)
• Redis
• Redis
• Redis

Progress is coming along, I recently [On the 15th] returned from a tour of France. There is a second period later in the summer I’ll have unknown-connectivity, but I’ll still be coding then.

After a week of working on the package management support as per my contract with Haiku, Inc., I have reached my first milestone: The base system is packaged... and it boots again. Here's a short account of what I have been doing exactly and what I'm up to next.

read more

My quarter term goals for the ZFS port included porting all the libzpool dependencies to Haiku. Out of four major dependencies -- libavl, libnvpair, libuutil and libumem -- I already have two -- libavl and libnvpair -- building on Haiku. libumem and libuutil will take another few days, which puts me at least a week behind my original schedule.

I'm currently working on porting libuutil, which is presenting a few roadblocks but nothing that can't be fixed in one day's work.

When I have some free time, I want to take a break from working on the port and do some cleanup. So far I've indiscriminately copied all the Solaris headers I need into my own repository. This is bad. I eventually want to use as many Haiku headers as possible, only importing those definitions from Solaris that are missing from the Haiku headers.

After all the libraries are working, the next step is porting libzpool itself.

My source repository is located at http://github.com/GeneralMaximus/zfs-haiku.

During the first few weeks of GSoC, I've gotten vboxsf, the shared folders module, mostly working, though I'm a bit behind where I'd like to be due to unexpected things repeatedly coming up both in physical-land and on my development setup. All supported functionality is working when accessed from the terminal - at this point I'm just optimizing and working on a bug where looking at a shared folder the wrong way in Tracker causes the kernel to panic. Mouse pointer integration, the other thing I had in my schedule for the first quarter, is now working; therefore finishing up the last few bits of vboxsf will be one of my main goals in the next few weeks.

The other goal I have now is time synchronization. It turns out this is more important than I thought - I didn't take into account how many things depend on timestamps being set to reasonable values, and my Haiku installation is at this point very confused as to what the current date and time are.

Several new features, including symlink support, were added in VBox 4, which I was completely unaware of until recently despite it having been released several months ago. I have now upgraded my laptop's copy of VBox so the Haiku port of the guest additions will be able to use these new features.

As anticipated, I was able to do little more than set up my environment during the first stretch of GSOC and the last few weeks of school and my previous internship. I was able to find computer with a compatible network and graphics card, which should hopefully be faster than running Haiku on a virtual machine on my normal computer. I also successfully installed haikuporter, although I am having some trouble using it. I am also trying out an IRC account.

I plan to install SDL 1.2 by the end of this week in order to make sure my environment is properly set up. I am currently having problems with haikuporter, which is complaining that the ports I wish to install do not exist. I'm not sure whether there is something else I need to download first or not to get this working; I posted a question about this on the HAIKU-GSOC mailing list. I hope to also make a stab at writing a .bep file for SDL 1.3; I admit I am not completely certain as to what this involves, but from reading the wiki it looks like it can be described as a python makefile?

Since school and my previous internship are completed, I should make much greater progress over the next few weeks. I look forward to finally being able to put my full time and effort into this project.

In my last blog post, I mentioned the following goals:

- Define an interface definition language
- Define preliminary bindings for a minimal test programy
- Write a preliminary generator to create the bindings
- Write the minimal test program

These have now been achieved, and the minimal test program (in Perl) runs. It shows a window with a button, and the button label changes when the button is clicked.

It still has problems, of course. For example, after 32 messages pass through MessagesReceived, it dies. But it's nice to see something actually running.

Anyone who would like to test it may check out the code at http://svn.osdrawer.net/perl-haiku-kits. There are instructions for generating and compiling the bindings in the repo.

The priority goals remaining for the first quarter are:
- Test threading issues
- Make a final choice on target languages

These additional goals (mentioned in the last post) may need to be postponed to the second quarter.
- Expand preliminary bindings and add new bindings
- Write test programs for the bindings
- Write documentation for the bindings

I was busy with finals throughout the Community Bonding period, which left me with little time to work on GSoC-related tasks. I still have 3 exams left with the last one being on June 7. That's when the fun starts. For now I'm merely playing with ZFS on FreeBSD on a virtual machine. I still need to make my way through at least the ZFS On-Disk Specification. Even though the information contained in this document is not strictly required for porting ZFS to Haiku, it's a useful read nonetheless. It also makes me look like a rockstar when I open it in coffee shops.

According to my proposal, this is what comes next:

• Finish porting dependencies (libavl, libnvpair, libuutil, libumem).
• Begin writing an OpenSolaris compatibility layer. This involves studying how threads, mutexes, condition variables etc. work on Solaris and then writing wrappers that behave in a similar manner around the Haiku API.

This means approximately one week spent on porting and testing dependencies and a fair bit of time spent on learning about low-level Solaris interfaces. One week is an optimistic estimation for a port of any kind, but some of the dependencies are merely libraries that implement useful data structures. This means they might build on Haiku without major changes. I can't say anything further unless I've taken a better look at them myself. The end result of the quarter term should be an OptionalPackage that contains all the external ZFS dependencies.

June 7 is not far :)

During the community bonding period, i have researched around the project to prepare my work for the coding days that will follow.
I also promised to talk with the other devs in the ml, it was not necessary in these days...i'm working with the help of Alex to a document describing the entire API in order to discuss it in the ml.

The first problem was to choose a Default Media Format for contact translators, my choice has been addressed to a flattened BMessage. BMessage will be used internally by BContact to represent the contact fields and the state of the object. BContact will be also a BArchivable object.

All that could look strange for a naive, but there are some aspects that should not be underestimated :

1. Easily sendable through applications and network (useful in future when sync support will come)
2. Make simple to save the state of an object on a disk and then restore it at the next boot

So one of the most important aspects of BMessage is their flexibility in storing informations. The contact kit will define a set of fields used to represent a generic contact in memory, but it's too restrictive! Apps and addons should be able to define custom types of fields since the API cannot in any case define a set of fields generic enough to support every existent and future API. This is already an obscure side for me and i need some hours to make clarity and choose the best solution.

Portable Contacts

Portable Contacts is REST API using OAuth with the aim to provide a generic access to contacts from different services, at the moment it support many services like the OpenSocial platform. Who has read my proposal probably remember something about a "Google Contacts Services Add-On". I have decided to switch it into a "Portable Contacts Addon". For more informations : http://portablecontacts.net/.

Goals For the first quarter

• A little patch for the translation kit to support the Contacts translators
• Basic BContact Implementation
• vCard and People translators

In the community bonding period I made some brief contacts with my two mentors and did some reading of specification for UVC devices as well as the header for the current USB kit. My mentors confirmed that the EHCI interface for isochronous transfers with USB devices is not working properly and does not have a good set of tests. Neither my camera nor my mentor's camera are successfully engaged by Codycam at the moment. Fixing this fundamental connection issue is the first priority for the first quarter of the GSoC period, before moving on to working on the UVC driver.

The GSoC is organized around a northern hemisphere summer and I am still finishing off my semester at university at the moment, so coding for Haiku hasn't been a possibility. A 100 page dissertation, final reports, projects and exams unfortunately take precedent at the moment. On June 7 I will have finished most major projects and only exams will be left. Exams shouldn't be a big deal, so I should be able to start development at that point and can hopefully make up for lost time.

I'll have more to report once I've gotten my hands dirty.

Some of the clearly amazing things that got done during the community bonding period were:

• Get Haiku logo
• Pasteboard (not good yet) Based heavily on examples in BeBook and guidance from Rene Gollent “DeadYak”
• command not found, trying to make Haiku’s bash more user friendly. Will be ready for Alpha 3.
• Haiku trove classifier TBA upon haiku-specific python packages!

I also discovered that Buildbot (Mostly its dependency: Twisted) don’t play nice on FreeBSD, the platform which currently builds Haiku nightlies on Matt Madia’s server. It’s not a huge priority to me at this point but it appears a bug still exists and will have to be filed (on my list).

On my list for the first quarter is finishing Command Not Found, and beginning work on the git tools and web services for package developers.[My actual project.] These tools can be developed while the package-fs work is in limbo or hpkgs mature. Command Not Found is a little script (if the install has python) or message that hooks into Bash’s command not found system and informs the user of ways to get this software. (i.e. you type in ruby but haven’t installed it or know that we have installoptionalpackage ruby, it will kindly tell you we have it.) Ubuntu has this feature in recent releases, and has an optional feature that may be interesting to use, spelling correction. It will be optional.

Command not found will be a way for users to be educated about Haiku even if they dive straight into the Terminal like a Unix user without reading any documentation (Like me!).

During the community bonding period I played around with the existing guest additions patch, getting it to build and switching my repository over to git to preserve my sanity. I've learned a lot about the way Haiku drivers and modules work, especially in the last few days, and it seems that a few things are simpler than I originally thought they'd be and some things are more difficult.

As an example of the latter, it turns out that drivers can't provide APIs to other drivers; only modules can do this. This posed a problem for the shared folders module, which needs to either be in a separate module or contain extreme hackiness. As a driver, the original vboxguest couldn't allow vboxsf to use its API. To solve this, mmu_man and I decided that the best first task would be to break up the existing vboxguest driver into a module (vboxguest) containing the guest additions and a driver (vboxdev) which exposes the library to userspace as /dev/misc/vboxguest. This is now finished.

It turns out guest properties and clipboard sharing are already working, and mouse pointer integration is at least partially implemented, though I've not managed to get it to work yet. This should free up a bit of time for some of the other things.

As my first major subproject, I'm planning to implement a shared folders filesystem module - as I (and possibly a couple of other students?) are coding exclusively in VirtualBox, it makes sense to focus on the things that make development easier first, and being able to share files with the host is certainly useful at times.

During the bonding period I looked into Python's extension tools; they seem to be straightforward and at first glance look relatively easy, so Python is definitely an option. I asked on the mailing list what other languages people were using on Haiku, and Neil kindly made a poll based on the results. Depending on the popularity of the language (based on the poll results) and the ease of writing extensions, I will make a final decision on which langauges to target during GSoC.

I also looked into various ways of defining the extensions. I have decided to use an interface definition and generate bindings from that. I looked at pidgen's IDL, but it didn't have all the necessary information. I also considered parsing the header files directly, but that would also lack some necessary information. I am currently working on defining an SGML-esque interface language. Using an SGML-like language means that if a new target needs information not currently contained in the interface definition, this information can be added without disrupting the parsers for existing targets. SGML also compresses nicely. I am, however, still open to suggestions for other solutions, since I haven't (yet) put enough time into this one to be irreversibly committed to it.

My goal for the first week is to get minimal functionality; I have selected Perl as the target for this portion because I already know how to write extensions for Perl. I am implementing enough of the Application, Message, Window, and Button objects to write a small test program. This should let me work out any issues with the interface language.

Once this test program is working, it should also allow me to test for thread issues. While I could deliberately write a program that blocked, what I'm looking for is situations where C++ would not block with equivalent code. As soon as the test program is working, I'm going to look at changing some data via the from the target language (running in an interpreter in the main thread) when called from Window::MessageReceived (running in the window's thread). If anyone has any other suggestions for creating blocking situations, let me know.

After I have made the final selection on target languages and determined the best way to avoid thread issues, I will continue by expanding the interface definitions used for the test program and creating new interface definitions for additional classes. I will also need to document the interface definition language and each of the classes. Class documentation, at least initially, will consist of the differences between the C++ interface described in the Be/Haiku Book and the interface for the target language in question, along with a link to the relevant page in the Be/Haiku Book.

Unfortunately, I haven't been able to stay online much so far. We have company visiting us, and they're staying in the room with the wireless router/modem. Since Haiku can't do encrypted wireless, and I can't use the only space physically close enough to the router to use an ethernet cable, I have to leave Haiku and boot into Windows whenever I want to use the internet. I tried running Haiku from the physical drive with VMware in Windows, but it's too slow. Does anyone know whether VirtualBox can use the physical drive, and if so, is it faster than VMware?

In summary, my goals for the first quarter are:
- Define an interface definition language
- Define preliminary bindings for a minimal test program
- Write a preliminary generator to create the bindings
- Write the minimal test program
- Test threading issues
- Make a final choice on target languages
- Expand preliminary bindings and add new bindings
- Write test programs for the bindings
- Write documentation for the bindings

During Community Bonding period, I began the process of setting up my Haiku working environment. Initially I attempted to install Haiku on my machine using VirtualBox; while this ran successfully, I could not access the internet or USB devices from my virtual machine (and hence could load the SDL libraries into Haiku). I have since created a Haiku-USB drive which also loads, although my graphics hardware breaks down if I do not run Haiku in safe mode, and Haiku still does not recognize my ethernet connection. I have also tried connecting via Wifi, with similar results.

I have briefly discussed the issues with my mentor, and will inform him and the Haiku community of my computer's hardware in a few days. For now all I can say is that it is a Sony VAIO (laptop), and I have had compatibility issues before with certain types of software and drivers. I will have access to another machine with different hardware after June 10, if necessary.

I hope to have a working Haiku installation on my machine, and possibly have downloaded and compiled SDL 1.2 by the quarter term.

===Edit 5/29/2011===
Thanks to my mentor and those who commented below, I was able to get internet on the VirtualBox working using the IntelPro 1000 chipset. This is slow, but it should be good enough for me to get started. Of course, since finals is the week after next, I will be doing almost no work during until June 10; but I will download SDL 1.2 onto the virtual machine and see if I can get it to compile.

In this lesson we seek to understand the part of the Interface Kit which lets us move things around in Tracker using the mouse. We will examine both ways of transferring information from one program to another, both the simple way and the more flexible (and complicated) method.

Programming with Haiku, Lesson 20

EDIT: 05/28/2011: Add card functionality as of r41792

I have recently been working on the radeon_hd graphics driver and accelerant to get extended mode setting complete for the Radeon r600-r800 chipsets (Roughly Radeon HD 31xx - Radeon HD 59xx)

We still have a *very* long way to go, however the following is now working in the driver:

• Identifying a pretty large range of Radeon HD cards based on PCIID
• Reading card information such as Memory and recording it
• Reading the active monitor EDID
• Creating mode lines from the EDID information above and adding them to the available mode lines
• Passing the active monitor EDID to the screen preflet for monitor vendor/model/serial identification

Here are the short-term todo items (with focus on getting extended mode setting working):

• Set non-vesa (extended) mode lines by toggling the vendor-specific Radeon registers
• Implement PLL controls
• Add a few card-specific screen preflet options

Here are the long-term todo items:

• Finish AtomBIOS support to support advanced card functionality
• Multi-monitor support

I am looking forward to (hopefully!) getting base functionality to the ATI Radeon HD cards. If you have and comments / PCI ID numbers / etc feel free to contact me and let me know.

Here are the current card test results as of 5/28/2011 r41792

Radeon HD 3450 r620 -- kallisti5 Primary monitor works at native resolution of 1440x900 but image offset and cut off. Radeon XT 2600 r630 -- Benjamin Angerer Primary monitor works perfectly. Plugging secondary monitor causes major issues as we are using EDID VESA results. (we try to use the VESA primary monitor modelines on the secondary monitor) Radeon HD 4350 r710 -- kallisti5 White screen of doom, incorrect screen mode change. Radeon HD 5830 r800 -- kallisti5 White screen of doom, incorrect screen mode change.