Monthly Archives: September 2014

Configure a Lubuntu 14.04 Development Image

I tend to follow a model of disposo-images for new development projects.  I create an image in ESXi for a project, put just the tools I want into the image, use it only for a month or two and then blow it away when I have either finished the project or simply want to return to a clean environment.  In the past I have used Ubuntu but most recently I have been using a lighter Lubuntu install and using X2Go for remoting the desktop.

The process below works for the 14.04 LTS release of Ubuntu.

Installing Lubuntu

The easiest way to install a minimal Lubunutu instance is to use the alternative installation process with the Ubuntu mini-iso distribution.  A link to the Lubuntu minimal install documentation appears below:

https://help.ubuntu.com/community/Lubuntu/Documentation/MinimalInstall

The mini-iso is small and the majority of the distribution is downloaded during the install process.  The installer is the standard Ubuntu text-graphical tool, not the more polished desktop installer.  The first screen appears below:

 

Lubuntu Install First Screen

First screen of the Ubuntu mini-iso installer.

Choose the ‘Install’ option and follow the dialogs.  I typically just choose to use the entire disk when prompted for storage options.  I also typically leave automatic updating off and handle that manually if I feel it makes sense to update.  Eventually, it will get to the Tasksel dialog.

Tasksel dialog.  Choose the OpenSSH server and the Lubuntu Desktop options.

Tasksel dialog. Choose the OpenSSH server and the Lubuntu Desktop options.

On this dialog I usually just choose OpenSSH and the Lubuntu Desktop.  The Lubuntu desktop doesn’t have nearly the bloat of the Ubuntu desktop so while it is not truly the ‘minimal installation’, it is very light none-the-less and you would have to install the majority of the pre-installed apps anyway.

If you are using ESXi you can install VMWare Tools at this point but you do not have to install them as most of the virtual drivers are already in the Ubuntu distribution and we will not be needing the nice X integration into the VSphere client.

Installing X2Go

X2Go is an implementation of the NX protocol and is based on the NX 3.x libraries as NoMachine went closed-source starting with the V4.0 libraries.  My experience with X2Go has been very good, the quality of the server and client are such that the projects have been adopted by the Fedora community.  Installing the X2Go server on Lubuntu is a breeze:

sudo apt-add-repository ppa:x2go/stable
sudo apt-get update
sudo apt-get install x2goserver x2goserver-xsession x2golxdebindings

Once installed, reboot the VM and you should be able to connect with an X2Go client.  As Lubuntu uses the LXDE desktop environment, it is necessary to configure the client session type as a ‘custom desktop’ with the following command:

lxsession -s Lubuntu -e LXDE

I typically map the desktop display to an entire monitor.  Sometimes on the first connection after a reboot, the desktop does not resize to fill the screen but I’ve found that suspending the session and then restarting it forces the display to resize.  Maybe there is a more elegant fix out there but the suspend/reconnect works anyway.

Installing Eclipse

I use Eclipse CDT as my C++ IDE.  Installation into a minimal Lubuntu environment is straightforward.  First, install a JRE or JDK and then just download the ‘Eclipse for C/C++ Developers’ zip file from the Eclipse website.  I use the OpenJDK 7 JDK

sudo apt-get install openjdk-7-jdk

After you have downloaded the Eclipse CDT zip file and extracted everything, it is usually nice to create a desktop shortcut.  You can do this in Lubuntu using the lxshortcut tool.  Open a terminal window and move to your ~/Desktop directory, then do the following.

cd ~/Desktop
lxshortcut -o eclipse

That will pop a small dialog, just click ‘OK’.  You should see a shortcut labelled ‘eclipse’ on your desktop.  Right-click on the icon and select ‘Shortcut Editor’ from the drop-down menu.  A dialog for the shortcut editor will open and will allow you to choose the executable for the shortcut (i.e. eclipse) and an icon file (i.e. icon.xpm from the eclipse install directory).

Installing the GCC C/C++ Compilers

The minimal install of Lubuntu does not include the GCC C/C++ compilers.  Adding them is also very straightforward.

sudo apt-get install build-essential

Conclusion

At this point, you should have an image with compilers, JDK and Eclipse CDT installed and configured.  I usually snapshot the image at this point so I can branch new, clean images for other projects without having to go through the install process again.

 

Building GCC Plugins – Part 3 C++ Libraries

As discussed in the prior post, I have started a set of C++ libraries to reduce the complexity of writing GCC Plugins and interpreting the GCC Abstract Syntax Tree.  In this post I will provide a high level description of the libraries and walk through the dependencies and directory structures.  The libraries are available on Github: ‘stephanfr/GCCPlugin’.

NB – At the time of writing, I am going through successive revisions and refactoring passes on the library, so expect anything you build now to break with my next commit to GitHub.  The interfaces will settle down in time and I will ‘chill’ them at some point in hopefully the not too distant future.

Licensing and Dependencies

All of the libraries with the exception of the unit test library link directly with the GCC source code, therefore they are all licensed with GPL V3.0.  The libraries are built with the C++11 language features and have dependencies on the Standard Library shipped with GCC and Boost libraries.  The unit testing framework depends on the Google Test libraries.

Programming Style

For what it is worth, I’ve been writing C++ code for a long, long time and am somewhat opinionated regarding some development practices.  First, I use anything in the standard c++ library – in particular I do not write containers.  Second, I use the std::string class in preference of char* strings almost exclusively.  For external interfaces I may expose a char* type but under the interface any char* will almost always map straight back to a std::string instance.  Third, I use anything from the Boost library that suits my needs.  The Boost libraries are excellent, don’t waste your time re-inventing a component in that library; in all likelihood your component will not be as good anyway.  Fourth, there are some naked pointers in these libraries but in general I try to use a std::unique_ptr or std::shared_ptr in any code written today (I will fix any naked pointers in this library as I refactor).  The standard library smart pointers are a bit more difficult to use than naked pointers, but that difficulty is a result of them enforcing the semantics necessary to know when to delete a pointer they wrap.  Finally, I really like C++ 11 – I’d strongly suggest cutting over to it.

With regard to my coding format, it is idiosyncratic.  Indentation and spacing don’t quite adhere to any standard, but at least I no longer use Hungarian notation – though that was a hard habit to kick.

Project and Directory Structure

The project is currently composed of seven directories, each with a single Eclipse CDT C++ project:

  1. CPPLanguageModel – a compiler-neutral class library of C++ language elements
  2. GCCInternalsTools – a set of classes and functions tailored specifically to the GCC g++ compiler to build a CPPLanguageModel representation of the code being compiled and to enable insertion of new code into the AST
  3. GCCInternalsUTFixture – a test fixture providing an abstraction of the GCCInternalsTools designed to permit the creation of unit tests for the library without any dependency on the GCC specific libraries themselves
  4. GCCInternalsUnitTest – a set of unit tests for key features of the GCC Plugins libraries
  5. TestExtensions – a collection of test framework ‘plugins’ that rely on GCCInternalsTools and the GCC headers; a separate project is used to prevent dependencies on GCC internals to leak into the main Unit Test framework
  6. GCCPlugin – a ‘HelloWorld’ style plugin for GCC Plugins using this framework
  7. Utility – Various utility classes to simply coding and implement design patterns I like

The most up-to-date examples of using the libraries will be in the unit test projects.  Similarly, if you go wandering through the code you will frequently see blocks of code commented out.  I tend to leave code I have refactored in place for a revision or two just in case a bug crawls out.  I find it is a bit easier than going back through prior revisions in source code control but it can make the code a little messy at points.  When I get to a version I am happy with, I go through a couple cleaning passes and knock out dead or legacy code.

Design Philosophy

The innards of GCC are absolutely not for the faint of heart.  A primary design goal of this framework is to insulate someone wanting to produce a GCC Plugin from the complexity of the compiler and its design paradigms.  At present, only a single GCC header file is required to build a plugin with this framework and all functionality exposed through the framework’s API is abstracted from GCC itself.  The framework is built for manipulating the Abstract Syntax Tree for C++ language programs but could be modified to match other languages.

To use the framework, you ought to only include header files from the CPPLanguageModel project.  Actually, the ASTDictionary.h and PluginManager.h header files will pull in most of the declarations needed to build your plugin.  Two header files from the gcc distribution are also needed: config.h and gcc-plugin.h

The object model exposed by the framework is that of a Dictionary of all the types and declarations in the code being compiled by g++ with the plugin loaded.  The dictionary is indexed by namespace, entry fully qualified name, entry source code location, entry UID and and an identity field.  All of the indices are exposed by the ASTDictionary class and can be used for searching the dictionary for a specific entry.  The identity, UID and fully qualified name indices are unique whereas the namespace and source location indices are non unique and may return a range of results.

The dictionary contains entries for different types and declarations.  Entries will be one of the following ‘kinds’: CLASS, UNION, FUNCTION, GLOBAL_VAR, TEMPLATE or UNRECOGNIZED.  The UNRECOGNIZED kind is simply a catch-all for any AST tree elements that have not yet been added to the tree parser.  Dictionary entries are effectively stubs from which the actual definition of the entry may be extracted.  Definitions contain the detailed, ‘kind’ specific information about the entry.  For example, the ClassDefinition object contains the base classes, fields, methods, template methods and friends for the class type.  Source location, namespace, UID, static and extern flags and a list of attributes are available for all dictionary entries and those values are copied into the more detailed definitions as well.

I’ve tried to insure that the AST tree parser will pass through the tree adding dictionary entries for elements it recognizes and ignoring everything else.  My intent is that it should not crash on encountering some language element it does not recognize in the AST but I have not run the parser over a whole lot of code so I will stick to ‘intent’ for now.  At present, the parser recognizes unions but does not yet provide a detailed definition of union types.  I figured it was more valuable to get some code injection functionality in place before sweating through the details of union representations in the GCC AST.

Current Supported Versions of GCC

The internals of GCC are constantly in flux and functionally there are no ‘frozen’ APIs or data structures that one can depend upon remaining static release over release.  The changes are unlikely to be significant release over release but there is a high probability of breaking changes associated with any release.

The code currently compiles and runs with GCC 4.8.0.  I can make no guarantees that it will compile and run with later releases, though hopefully nothing should break between double dot releases.

Example Plugin

An example ‘HelloWorld’ plugin appears below.  The four header files appear at the top.  The plugin_is_GPL_compatible symbol is needed for licensing compliance with the GCC suite.

There exists an implementation of the CPPModel::CallbackIfx interface which is used by the framework to call back into the plugin at specific times in the compilation process.  There are entry points for when the AST is ready, for a point at which namespaces may be declared and a point at which code may be injected.  For the sample plugin, all that happens is that the contents of the TestNamespace inside the code being compiled is dumped to cerr.  The plugin_init function is part of the GCC plugin framework and is rather straightforward when using these abstraction libraries.


/*-------------------------------------------------------------------------------
Copyright (c) 2013 Stephan Friedl.

All rights reserved. This program and the accompanying materials
are made available under the terms of the GNU Public License v3.0
which accompanies this distribution, and is available at
http://www.gnu.org/licenses/gpl.html

Contributors:
 Stephan Friedl
-------------------------------------------------------------------------------*/

#include "config.h"

#include "ASTDictionary.h"
#include "PluginManager.h"

#include "gcc-plugin.h"

int plugin_is_GPL_compatible;

class Callbacks : public CPPModel::CallbackIfx
{
public :

 Callbacks()
 {}

 virtual ~Callbacks()
 {}

 void ASTReady()
 {
 std::list<std::string> namespacesToDump( { "TestNamespace::" } );

 CPPModel::GetPluginManager().GetASTDictionary().DumpASTXMLByNamespaces( std::cerr, namespacesToDump );
 };

 void CreateNamespaces()
 {
 };

 void InjectCode()
 {
 };

};

Callbacks g_pluginCallbacks;

int plugin_init( plugin_name_args* info, plugin_gcc_version* ver )
{
 std::cerr << "Starting Plugin: "<< info->base_name << std::endl;

 CPPModel::GetPluginManager().Initialize( "HelloWorld Plugin", &g_pluginCallbacks );

 return( 0 );
}

 

Example Output

A sample program to be compiled appears below.  This code has the TestNamespace declared and it is the contents of that namespace that will be dumped by the plugin above.

#include <iostream>

namespace TestNamespace
{
	class TestClass
	{
	public :

		int			publicInt;

		int			getPublicInt() const
		{
			return( publicInt );
		}

	protected :

		double		getPrivateDouble() const
		{
			return( privateDouble );
		}

	private :

		double		privateDouble;
	};

	char*		globalString = "This is a global string";

	TestClass	globalTestClassInstance;
}

int main()
{
	std::cout << "!!!Hello World!!!" << std::endl; // prints !!!Hello World!!!

	return 0;
}

The command line required to invoke g++ with the plugin and compile the above file follows:

/usr/gcc-4.8.0/bin/gcc-4.8.0 -c -std=c++11 -fplugin=libGCCPlugin.so HelloWorld.cpp

When g++ initializes, it loads the sample plugin and when the AST is ready, the plugin dumps the following to the standard output.  It isn’t prefect XML but ought to be good enough to analyze the program being compiled.

8: 2014-09-23 21:03:42   [LoggingInitialization] [NORMAL]  Logging Initiated
Starting Plugin: libGCCPlugin
HelloWorld.cpp:34:24: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
  char*  globalString = "This is a global string";
                        ^
<ast>
    <dictionary>
        <namespace name="TestNamespace::">
            <dictionary_entry>
                <namespace>
                    <name>TestNamespace::</name>
                </namespace>
                <name>TestClass</name>
                <uid>20720</uid>
                <source-info>
                    <file>HelloWorld.cpp</file>
                    <line>9</line>
                    <char-count>1</char-count>
                    <location>6451683</location>
                </source-info>
            </dictionary_entry>
            <dictionary_entry>
                <namespace>
                    <name>TestNamespace::</name>
                </namespace>
                <name>globalString</name>
                <uid>28506</uid>
                <source-info>
                    <file>HelloWorld.cpp</file>
                    <line>34</line>
                    <char-count>1</char-count>
                    <location>6454884</location>
                </source-info>
                <static>true</static>
            </dictionary_entry>
            <dictionary_entry>
                <namespace>
                    <name>TestNamespace::</name>
                </namespace>
                <name>globalTestClassInstance</name>
                <uid>28507</uid>
                <source-info>
                    <file>HelloWorld.cpp</file>
                    <line>36</line>
                    <char-count>1</char-count>
                    <location>6455143</location>
                </source-info>
                <static>true</static>
            </dictionary_entry>
        </namespace>
    </dictionary>
    <elements>
        <namespace name="TestNamespace::">
            <class type="class">
                <name>TestClass</name>
                <uid>20720</uid>
                <source-info>
                    <file>HelloWorld.cpp</file>
                    <line>9</line>
                    <char-count>1</char-count>
                    <location>6451683</location>
                </source-info>
                <namespace>
                    <name>TestNamespace::</name>
                </namespace>
                <compiler_specific>
                    </artificial>
                </compiler_specific>
                <base-classes>
                </base-classes>
                <friends>
                </friends>
                <fields>
                    <field>
                        <name>publicInt</name>
                        <source-info>
                            <file>HelloWorld.cpp</file>
                            <line>13</line>
                            <char-count>1</char-count>
                            <location>6452196</location>
                        </source-info>
                        <type>
                            <kind>fundamental</kind>
                            <declaration>int</declaration>
                        </type>
                        <access>PUBLIC</access>
                        <static>false</static>
                        <offset_info>
                            <size>4</size>
                            <alignment>4</alignment>
                            <offset>0</offset>
                            <bit_offset_alignment>128</bit_offset_alignment>
                            <bit_offset>0</bit_offset>
                        </offset_info>
                    </field>
                    <field>
                        <name>privateDouble</name>
                        <source-info>
                            <file>HelloWorld.cpp</file>
                            <line>30</line>
                            <char-count>1</char-count>
                            <location>6454374</location>
                        </source-info>
                        <type>
                            <kind>fundamental</kind>
                            <declaration>double</declaration>
                        </type>
                        <access>PRIVATE</access>
                        <static>false</static>
                        <offset_info>
                            <size>8</size>
                            <alignment>8</alignment>
                            <offset>0</offset>
                            <bit_offset_alignment>128</bit_offset_alignment>
                            <bit_offset>64</bit_offset>
                        </offset_info>
                    </field>
                </fields>
                <methods>
                    <method>
                        <name>getPublicInt</name>
                        <uid>28497</uid>
                        <source-info>
                            <file>HelloWorld.cpp</file>
                            <line>15</line>
                            <char-count>1</char-count>
                            <location>6452452</location>
                        </source-info>
                        <access>PUBLIC</access>
                        <static>false</static>
                        <result>
                            <type>
                                <kind>fundamental</kind>
                                <declaration>int</declaration>
                            </type>
                        </result>
                        <parameters>
                            <parameter>
                                <name>this</name>
                                <type>
                                    <kind>derived</kind>
                                    <declaration>
                                        <operator>pointer</operator>
                                        <type>
                                            <kind>class-or-struct</kind>
                                            <declaration>TestNamespace::TestClass</declaration>
                                            <namespace>
                                                <name>TestNamespace::</name>
                                            </namespace>
                                        </type>
                                    </declaration>
                                </type>
                                <compiler_specific>
                                    </artificial>
                                </compiler_specific>
                            </parameter>
                        </parameters>
                    </method>
                    <method>
                        <name>getPrivateDouble</name>
                        <uid>28499</uid>
                        <source-info>
                            <file>HelloWorld.cpp</file>
                            <line>22</line>
                            <char-count>1</char-count>
                            <location>6453350</location>
                        </source-info>
                        <access>PROTECTED</access>
                        <static>false</static>
                        <result>
                            <type>
                                <kind>fundamental</kind>
                                <declaration>double</declaration>
                            </type>
                        </result>
                        <parameters>
                            <parameter>
                                <name>this</name>
                                <type>
                                    <kind>derived</kind>
                                    <declaration>
                                        <operator>pointer</operator>
                                        <type>
                                            <kind>class-or-struct</kind>
                                            <declaration>TestNamespace::TestClass</declaration>
                                            <namespace>
                                                <name>TestNamespace::</name>
                                            </namespace>
                                        </type>
                                    </declaration>
                                </type>
                                <compiler_specific>
                                    </artificial>
                                </compiler_specific>
                            </parameter>
                        </parameters>
                    </method>
                </methods>
                <template_methods>
                </template_methods>
            </class>
            <global_var_entry>
                <namespace>
                    <name>TestNamespace::</name>
                </namespace>
                <name>globalString</name>
                <uid>28506</uid>
                <source-info>
                    <file>HelloWorld.cpp</file>
                    <line>34</line>
                    <char-count>1</char-count>
                    <location>6454884</location>
                </source-info>
                <static>true</static>
                <type>
                    <kind>derived</kind>
                    <declaration>
                        <operator>pointer</operator>
                        <type>
                            <kind>fundamental</kind>
                            <declaration>char</declaration>
                        </type>
                    </declaration>
                </type>
            </global_var_entry>
            <global_var_entry>
                <namespace>
                    <name>TestNamespace::</name>
                </namespace>
                <name>globalTestClassInstance</name>
                <uid>28507</uid>
                <source-info>
                    <file>HelloWorld.cpp</file>
                    <line>36</line>
                    <char-count>1</char-count>
                    <location>6455143</location>
                </source-info>
                <static>true</static>
                <type>
                    <kind>class-or-struct</kind>
                    <declaration>TestNamespace::TestClass</declaration>
                    <namespace>
                        <name>TestNamespace::</name>
                    </namespace>
                </type>
            </global_var_entry>
        </namespace>
    </elements>
</ast>
Declaring Globals

Conclusion

It has taken a while to get this far but I will dive into the internals of the framework and provide examples of code injection in future posts.

 

Printing with Taulman Bridge Nylon

I have been dabbling in 3D printing for the last six months with my MakerGear M2 printer (http://www.makergear.com a fantastic precision machine tool) and have done a lot of printing in PLA.  PLA is great for a lot of objects, particularly the various elephants and other things I print for my kids but is too brittle for some types of applications.  I have a couple projects I am contemplating that require a tougher material, so I gave Taulman Bridge Nylon a try.  Like all things in 3D printing, there is a learning curve but once you have a process for printing with this nylon, it is a fantastic material.

Taulman Bridge

The ‘Bridge’ nylon is intended to combine the toughness of nylon with the printing ease of PLA.  I have not tried printing with regular nylon so I cannot comment on how much easier it is to print with Bridge, but printing with Bridge is not quite like printing with PLA.  Bridge still absorbs water, it requires a higher print temperature and at least for me, I need to print with thicker layers at a slower speed with Bridge than with PLA.  That said, with the right adjustments in place my success printing with Bridge is pretty darn close to my success rate with PLA – probably 90%+ prints I start complete acceptably.

Using Bridge

First off, this is the process I use.  I live in Colorado at about 5000ft elevation and relatively low humidity.  Your mileage may vary…

Step 1: Dry the Material

Bridge comes on a small diameter spool sealed in a bag with silica gel drying packets inside the bag.  Despite the Bridge formulation to reduce wetness, the packaging and the relative dryness of Colorado, I had little success printing with Bridge right out of the bag.  Right out of the bag, you will see a lot of steam coming out of the nozzle and I got intermittent sputtering as well.  I tried baking the spool in the oven for 6 hours at 175F and this seemed to work reasonably well though the spool deformed a bit.  Also, I got the sense that the inner layers of the spool may not have dried as well as the outer layers.

I adjusted my drying technique a bit by getting a small toaster oven from Walmart and then using it to bake just enough loose filament pulled off the spool for a given print:

Baking Taulman Bridge Nylon

Preparing to dry a length of Taulman Bridge nylon in a small toaster oven in my garage. I typically pull enough material from the main spool for a specific print, clip it and then bake it loose in the oven at 175F for 6 to 8 hours.

After baking, I let the material cool in the oven for a bit and then I transfer it immediately into a zip-lock bag with a couple packs of silica gel for further cooling and drying overnight.

Step 2:  Printer Settings

I use the Simplify3D (http://www.simplify3d.com) software package to slice my models and control the M2.  I’ve used a number of the more popular open source packages as well but I really like all the key functions in one place with an easy to navigate GUI.  I took the suggested settings for the M2 and nylon and through trial and error made some adjustments from that starting point.  The primary problem I ran into was ‘popcorn’ from the print instead of a smooth stream of nylon.  After that I also had some adhesion and warping issues.  The four main changes I made were to bump up the extruder temperature to 245C, start the build plate at 70C then ramp to 90C for the second layer on, cut the printing speed in half and finally stick to a 0.3mm layer height.

Below are the config screens from Simplify3D with the settings I use for Bridge:

Taulman-Extruder

Extruder settings for printing with Taulman Bridge on my MakerGear M2. Note the Ooze Control settings.

Taulman-Layer

Layer settings

Taulman-PrimaryExtruder

Extrusion temperature set at 245C

Taulman-HeatedBed

Heated bed at 70C for the first layer and then 90C for the rest.

Taulman-Other

I reduced the default printing speed by 50% from the speed used for PLA or suggested for nylon,

Taulman-Advanced

Retraction control and ooze rate.

Step 3: Preparing the Build Plate

A couple test prints with clean glass had adhesion and/or warping problems.  I gave the Elmer’s glue coating I use for PLA a shot and it worked extraordinarily well.  So well in fact that I have to use a very thin layer of glue, as with thicker layers the nylon is very, very hard to detach.  Below are a couple picture of the wet glue on the plate and what it looked like dry just before printing.

Build Plate with Fresh Elmer's Glue

Build plate with a thin coating of Elmer’s white glue to improve nylon adhesion.

Build Plate with Dried Elmer's Glue

Plate with the dried glue at 70C.

Finished Product:

For an example of what is possible with Bridge on the M2, I printed Emmett’s Gear Bearing from Thingiverse (http://www.thingiverse.com/thing:53451).  This is an absolutely ingenious design of a bearing that can only be produced with 3D printing.  If you wanted to use this in a real project, then nylon would be a far better material than PLA or ABS.  If you look at the design, it is pretty clear that if your printer or material aren’t dialed in well your odds of getting a working bearing are slim.  There are lots of opposing surfaces which could fuse and render the bearing a hockey puck.  Emmett’s designs on Thingiverse are exceptional, if you have not looked them over then do yourself a favor and do so.

If the piece will not come loose easily from the build plate, I usually put the plate in the freezer for 5 or 10 minutes after which the piece generally pops right off.  Check the start of the video for that demo.

Completed Bearing in Nylon

Completed Gear Bearing by Emmett printed on a MakerGear M2 with Taulman Bridge nylon.

Demo Video:

 

Final Thoughts :

My experience with the Bridge material has been great, once I got the process right.  Dry material, higher temperature, slow printing, thick layers and glue for adhesion all seem to matter but the results as demonstrated by the pictures and video are pretty self-evident.