Deploying COSMOS to the Cloud

COSMOS is a GUI application which runs equally well on Windows, Linux, and Mac OS X due to the QT GUI framework and Ruby language it is written in. Traditionally this means you install it locally on your own workstation and you’re off and running. But can COSMOS also be deployed to the cloud? Yes! This post describes how I deployed COSMOS to Amazon Web Services (AWS) using several different technologies.

AWS

AWS consists of a lot of different services. To deploy COSMOS you need to create an AWS EC2 instance. The first step is to sign up for AWS which enables you to use their Free Tier for 12 months. This includes 750hrs each on Linux and Windows Server.

Deploying to Windows Server

Deploying to Windows Server is probably the easiest way to get COSMOS in the cloud. Simply create a Windows Server instance by selecting the following image: Windows Image

Then select the t2.micro Type which is marked “Free tier eligible”. Launch the Instance and you should see the key pair dialog: Key Pair Dialog

Create a new key pair and give the name something generic because you can use the same key pair for all the EC2 instances you create. Create the instance and then View Instance which will show the instance status. Click the Connect button at top which will bring up the Connect dialog: Connect Dialog

Click the Download Remote Desktop File and open it in Remote Desktop to connect to the Windows Server instance. Note that it does take a while for the Windows instance to boot so this won’t work until your Status Checks show a green check. Also note that many corporate firewalls may block doing a Remote Desktop outside your corporate network.

You also need to click Get Password and locate your ‘pem’ file you saved earlier to Decrypt your password. Login to the instance as Administrator with the decrypted password. Once you’ve logged in you can change the password to something a little more reasonable. Then simply follow the usual COSMOS installation instructions.

Here is a screenshot of my successful COSMOS installation running on the AWS Microsoft Server instance: COSMOS on Windows

Deploying to Red Hat Linux (with X forwarding)

Deploying to Red Hat Linux is similar to Windows. Create a Red Hat instance by selecting the following image: Red Hat Image

Use the same key pair when creating your Windows instance and create the instance. View Instance and click the Connect button which brings up the Connect dialog: Connect Dialog

SSH to the instance using the connection string provided making sure to specify the full path to your ‘pem’ file in the quoted path after the -i option. Install a GUI by issuing the following command:

sudo yum groupinstall 'Server with GUI'

Install COSMOS using the installation bash file:

bash <(\curl -sSL https://raw.githubusercontent.com/BallAerospace/COSMOS/master/vendor/installers/linux_mac/INSTALL_COSMOS.sh)

Chose the sudo option when asked how to install. To enable X forwarding edit the SSH config file:

sudo vim /etc/ssh/sshd_config

Enable the following settings:

X11Forwarding yes
X11DisplayOffset 10
X11UseLocalhost no

Close your current SSH connection and reconnect with SSH adding the -X option to enable X11 forwarding and -Y to enable trusted X11 fowarding. If you host OS is Mac OS X you’ll need to install XQuartz. Linux has X11 forward support built-in. Windows should probably install Xming which is an exercise left to the reader.

Now launch COSMOS and you should see the COSMOS windows appear on your own machine. While this approach works I found the performance to be significantly slower than VNC. Here is a screenshot of it running while I connected via a Mac OS X machine: Red Hat X

Deploying on Ubuntu (with VNC)

Deploying to Ubuntu Linux is very similar to Red Hat. Create an Ubuntu instance by selecting the following image: Ubuntu Image

Use the same key pair as when creating your Windows or Red Hat instance and create the instance. View Instance and click the Connect button which brings up the Connect dialog: Connect Dialog

Click on the instance and click the Description tab which appears below the instance. Click the link next to Security groups to open the Security Groups configuration. Click the Inbound tab and create Edit to create a new Custom TCP Rule to enable TCP traffic on Ports 5900-5910 from Anywhere. Your rule should look like the following: Security Group

SSH to the instance using the connection string provided making sure to specify the full path to your ‘pem’ file in the quoted path after the -i option. Install a GUI by issuing the following command:

sudo apt-get update
sudo apt install xfce4 xfce4-goodies tightvncserver

Run the following commands to setup the VNC server:

vncserver
vncserver -kill :1
vim ~/.vnc/xstartup

Ensure your xstartup file matches the following:

#!/bin/sh

# Uncomment the following two lines for normal desktop:
unset SESSION_MANAGER
# exec /etc/X11/xinit/xinitrc
unset DBUS_SESSION_BUS_ADDRESS
startxfce4 &

[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
#x-terminal-emulator -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &
#x-window-manager &

Restart VNC:

vncserver

Install COSMOS using the installation bash file:

bash <(\curl -sSL https://raw.githubusercontent.com/BallAerospace/COSMOS/master/vendor/installers/linux_mac/INSTALL_COSMOS.sh)

Chose the sudo option when asked how to install. On your local machine install a VNC viewer such as TightVNC and connect by entering the Public DNS address of your AWS instance in the Remote Host as well as the Port number of 5901. Typically this is added by appending it to the Remote Host address with a colon. Here is a screenshot of it running while I connected via TightVNC: Red Hat X

If you need additional support please contact us at cosmos@ball.com.

Ball Aerospace COSMOS 4.0.1 Released

New Features:

  • #527 Editing config files should now bring up ConfigEditor
  • #528 ConfigEditor missing some keywords
  • #534 Create ConfigEditor Mac app
  • #536 Clickable canvas objects open screens
  • #542 Automatically populate COMMAND SYSTEM META
  • #543 Allow SYSTEM META items to be read only

Maintenance:

  • None

Bug Fixes:

  • #533 TestRunner strips all comments when running
  • #538 META_INIT broken
  • #540 Background task packet subscription get_packet broken
  • #547 convert_packet_to_data should copy buffer

Migration Notes from COSMOS 3.8.x:

COSMOS 4 includes several breaking changes from the COSMOS 3.x series.

The first and simplest is that the Command and Telemetry Server now opens an additional port at 7780 by default, that provides a router that will send out each command that the system has sent. This can allow external systems to also log all commands sent by COSMOS. For most people this change will be transparent and no updates to your COSMOS configuration will be required.

The second is that the Command and Telemetry Server now always supports a meta data packet called SYSTEM META. This packet will always contain the MD5 sum for the current running COSMOS configuration, the version of COSMOS running, the version of your COSMOS Project, and the version of Ruby being used. You can also add your own requirements for meta data with things like the name of the operator currently running the system, or the name of a specific test you are currently running. In general you shouldn’t need to do anything for this change unless you were using the previous metadata functionality in COSMOS. If you were, then you will need to migrate your meta data to the new SYSTEM META packet, and change the parameters in your CmdTlmServer or TestRunner configurations regarding meta data. If you weren’t using metadata before, then you will probably just notice this new packet in your log files, and in your telemetry stream.

Finally the most exciting breaking change is in how COSMOS interfaces handle protocols. Before, the COSMOS TCP/IP and Serial interface classes each took a protocol like LENGTH, TERMINATED, etc that defined how packets were delineated by the interface. Now each interface can take a set of one or more protocols. This allows COSMOS to much more easily support nested protocols, such as the frame focused protocols of CCSDS. It also allows for creating helpful reusable protocols such as the new CRC protocol for automatically adding CRCs to outgoing commands and verifying incoming CRCs on telemetry packets. It’s a great change, but if you have any custom interface classes you have written, they will probably require some modification. See the Interfaces section at cosmosrb.com to see how the new interface classes work. We will also be writing up a blog post to help document the process of upgrading. Look for this in a week or two.

To upgrade to the latest version of COSMOS, run “bundle update cosmos” in your COSMOS project folder.

Ball Aerospace COSMOS 4.0.0 Released

COSMOS 4 is here!

48 tickets have gone into this release, and it brings with it two new tools and some great under the hood improvements.

New Tools:

COSMOS now has a dedicated Configuration Editor and Command Sequence Builder.

The config editor gives you contextual help when building config files, and make it super easy to define packets and configure tools without having to have the online documentation up in front of you. It’s going to make setting up COSMOS even easier than it was before.

Command Sequence builder allows you to define series of commands that should be sent at either absolute or relative timestamps to each other. This is great for planning time specific commanding. You can execute these on the ground directly from the tool, or you can convert them to your own internal format and upload to the system you are commanding.

Highlighted changes:

  1. New protocol system allows assigning multiple protocols to each interface to support layered protocols, and common functionality like CRC checking/adding to commands.
  2. The ability to View the most recent raw data received or sent on each interface
  3. The Command and Telemetry Server can now run on JRuby in –no-gui mode (may help performance for huge projects with 50+ targets)
  4. New router provides the ability to get a copy of every command sent out from COSMOS in a stream
  5. New SYSTEM META packet output by the CmdTlmServer
  6. Lots more! See the full ticket list below

New Features:

  • #229 Gem Based Targets should support DataViewer and other tool configurations
  • #234 Add a method to system.txt to add files used in the marshall file MD5 sum calculation
  • #253 Create “generators” for targets, tools, etc
  • #258 Create COSMOS Command Sequence Tool
  • #261 Provide a method for specifying binary data in STRING and BLOCK default values
  • #278 Consider adding wait_ methods to internal API for use in Background tasks
  • #281 Add support for stretch and spacers in widget layouts
  • #319 Add the ability to grab telemetry ARRAY_ITEMs
  • #337 Support specifying default parameters to default log reader and log writer in system.txt
  • #347 COSMOS GLobal Time Zone Setting (Local/UTC)
  • #356 Interface protocols
  • #360 Add raw stream preamble and postamble data to “View Raw”
  • #381 Float Infinity and NaN as command values
  • #401 tolerance scripting calls should support array telemetry items
  • #404 Packet Viewer easy access to edit configuration files
  • #405 Telemetry Viewer easy access to edit screen definition
  • #423 Add “Cmd router” to CmdTlmServer to support external logging of all commands
  • #424 TlmViewer should call update_widget for a screen with no value items and with CmdTlmServer not running
  • #426 Standardize meta data to SYSTEM META packet
  • #432 Export processed config files
  • #442 Label value widgets should support right aligned labels
  • #459 Script Editor code completion enhancements
  • #479 Limits Monitor doesn’t detect newly connected targets
  • #489 Built in support for limits group enable and disable
  • #497 Update serial_interface.rb to support hardware flow control
  • #498 Script helper for activities that should cause an exception
  • #511 Make CmdTlmServer run on JRuby
  • #512 Create a CRC Protocol
  • #513 Create a GUI config file editor
  • #516 Recreate COSMOS C Extension Code in Pure Ruby
  • #517 Make hostname for tools to connect to CTS API configurable in system.txt
  • #519 Replay should support alternate packet log readers

Maintenance:

  • #354 Targets need to be path namespaced to avoid conflicts
  • #323 Catch Signals in CmdTlmServer
  • #341 Document COSMOS JSON API on cosmosrb.com
  • #398 Documentation, code cleanup
  • #429 Command Endianness and Parameter Endianness
  • #437 Remove CMD_TLM_VERSION from system.txt
  • #438 Cache script text as part of instrumenting script
  • #446 Windows 10 Install fails
  • #476 Separate apt and yum package install lines
  • #477 Deprecate userpath.txt
  • #484 require_file should re-raise existing exception

Bug Fixes:

  • #456 Replay doesn’t shut down properly if closed while playing
  • #481 show_backtrace not working in ScriptRunner
  • #494 Details dialog crashes for items with LATEST packet
  • #502 Target REQUIRE should also search system path
  • #506 Don’t call read_interface if data is cached in protocols for another packet

Migration Notes from COSMOS 3.8.x:

COSMOS 4 includes several breaking changes from the COSMOS 3.x series.

The first and simplest is that the Command and Telemetry Server now opens an additional port at 7780 by default, that provides a router that will send out each command that the system has sent. This can allow external systems to also log all commands sent by COSMOS. For most people this change will be transparent and no updates to your COSMOS configuration will be required.

The second is that the Command and Telemetry Server now always supports a meta data packet called SYSTEM META. This packet will always contain the MD5 sum for the current running COSMOS configuration, the version of COSMOS running, the version of your COSMOS Project, and the version of Ruby being used. You can also add your own requirements for meta data with things like the name of the operator currently running the system, or the name of a specific test you are currently running. In general you shouldn’t need to do anything for this change unless you were using the previous metadata functionality in COSMOS. If you were, then you will need to migrate your meta data to the new SYSTEM META packet, and change the parameters in your CmdTlmServer or TestRunner configurations regarding meta data. If you weren’t using metadata before, then you will probably just notice this new packet in your log files, and in your telemetry stream.

Finally the most exciting breaking change is in how COSMOS interfaces handle protocols. Before, the COSMOS TCP/IP and Serial interface classes each took a protocol like LENGTH, TERMINATED, etc that defined how packets were delineated by the interface. Now each interface can take a set of one or more protocols. This allows COSMOS to much more easily support nested protocols, such as the frame focused protocols of CCSDS. It also allows for creating helpful reusable protocols such as the new CRC protocol for automatically adding CRCs to outgoing commands and verifying incoming CRCs on telemetry packets. It’s a great change, but if you have any custom interface classes you have written, they will probably require some modification. See the Interfaces section at cosmosrb.com to see how the new interface classes work. We will also be writing up a blog post to help document the process of upgrading. Look for this in a week or two.

To upgrade to the latest version of COSMOS, run “bundle update cosmos” in your COSMOS project folder.

Template Protocol

The Template Stream Protocol is probably one of the more confusing protocols in the COSMOS stream protocol library but it is extremely helpful when implementing string based protocols such as Standard Commands for Programmable Instruments (SCPI; often pronounced “skippy”).

For this example we’ll assume we’re trying to talk to a SCPI enabled power supply such as the Keysight N6700. We start by creating a directory under our config/targets called POWER. The supply has a TCP/IP interface so we’ll use the TCP/IP Client Interface to connect to it. Thus we create our POWER/cmd_tlm_server.txt file as follows:

INTERFACE POWER_INT tcpip_client_interface.rb 127.0.0.1 5025 5025 10.0 nil TEMPLATE 0x0A 0x0A
  TARGET POWER

This definition declares an interface named POWER_INT using the TCP/IP client interface which connects to ‘127.0.0.1’ (obviously you’ll change this to your actual power supply IP addres) using a write and read port of 5025 (standard SCPI ports for Keysight instruments) with a write timeout of 10s and no read timeout (block on read). We specify the TEMPLATE protocol with both write and read termination characters of 0x0A (ASCII newline). Note the TEMPLATE protocol takes many additional parameters to allow you to work with off nominal protocol conditions.

Now you can define your target’s command and telemetry definitions. We’ll create example commands which get and set the voltage setting in our power supply. Create a POWER/cmd_tlm/cmd.txt file which has the following:

COMMAND POWER GET_VOLTAGE BIG_ENDIAN "Get voltage"
  APPEND_ID_PARAMETER CMD_ID 8 UINT 1 1 1 "Command Id" # Unique command ID
  APPEND_PARAMETER CHANNEL 8 UINT 1 4 1 "Channel"
  APPEND_PARAMETER CMD_TEMPLATE 512 STRING "MEAS:VOLT? (@<CHANNEL>)"
  APPEND_PARAMETER RSP_TEMPLATE 512 STRING "<VOLTAGE>"
  APPEND_PARAMETER RSP_PACKET 512 STRING "TLM"

COMMAND POWER SET_VOLTAGE BIG_ENDIAN "Set voltage"
  APPEND_ID_PARAMETER CMD_ID 8 UINT 2 2 2 "Command Id" # Unique command ID
  APPEND_PARAMETER CHANNEL 8 UINT 1 4 1 "Channel"
  APPEND_PARAMETER VOLTAGE 8 UINT 0 100 10 "Voltage"
  APPEND_PARAMETER CMD_TEMPLATE 512 STRING "VOLT <VOLTAGE>,(@<CHANNEL>)"
  APPEND_PARAMETER RSP_TEMPLATE 512 STRING "<SET_VOLTAGE>"
  APPEND_PARAMETER RSP_PACKET 512 STRING "TLM"

The CMD_ID parameter is defined by APPEND_ID_PARAMETER. This ID parameter is not used by the SCPI protocol but is needed for COSMOS to identify the command when it is logged. The CMD_TEMPLATE parameter is the actual SCPI command which is being sent to the target. Anything inside brackets <> will be replaced by the value in the named parameter. For example, both commands define the CHANNEL parameter and thus <CHANNEL> will be replaced by the value of that parameter when constructing the command. The RSP_TEMPLATE is the expected string response back from the target. This is parsed by pulling out values into the bracket delimited values. The RSP_PACKET defines the packet where the bracket delimited values are defined. So for our GET_VOLTAGE example we parse the VOLTAGE value and place it in the TLM packet.

Create a POWER/cmd_tlm/tlm.txt file to define the response telemetry:

TELEMETRY POWER TLM BIG_ENDIAN "Power Supply Telemetry"
  APPEND_ID_ITEM TLM_ID 32 INT 1 "Packet Identifier" # Unique telemetry ID
  APPEND_ITEM VOLTAGE 32 FLOAT "PS Measured Voltage"
    FORMAT_STRING "%0.3f"
    UNITS "Volts" "V"
  APPEND_ITEM SET_VOLTAGE 32 FLOAT "PS Set Voltage"
    FORMAT_STRING "%0.3f"
    UNITS "Volts" "V"

The TLM_ID item is defined by APPEND_ID_ITEM. This ID item is not used by the SCPI protocol but is needed by COSMOS to decode this logged telemetry packet. The packet is named TLM which matches our RSP_PACKET definition in the commands. We define VOLTAGE and SET_VOLTAGE which also match the values used in the RSP_TEMPLATE parameters in our commands.

With Keysight supplies you can string together a bunch of SCIP commands in one CMD_TEMPLATE if you delimit them with semicolons. Then in the RSP_TEMPLATE you can break the response apart and set a bunch of telemetry items at once. For example:

COMMAND POWER GET_STATUS BIG_ENDIAN "Get status"
  APPEND_ID_PARAMETER CMD_ID 8 UINT 3 3 3 "Command Id" # Unique command ID
  APPEND_PARAMETER CHANNEL 8 UINT 1 4 1 "Channel"
  APPEND_PARAMETER CMD_TEMPLATE 512 STRING "MEAS:VOLT (@<CHANNEL>);CURR (@<CHANNEL>);POW (@<CHANNEL>)"
  APPEND_PARAMETER RSP_TEMPLATE 512 STRING "<VOLTAGE>,<CURRENT>,<POWER>"
  APPEND_PARAMETER RSP_PACKET 512 STRING "TLM"

The RSP_TEMPLATE expects to have three values delimited by the comma character. For this example to be complete you would also need to declare CURRENT and POWER items in the TLM packet.

Using the TEMPLATE processor can be complex but makes working with with string based command / response protocols like SCPI much easier. If you need additional support please contact us at cosmos@ball.com.

Ball Aerospace COSMOS 3.9.2 Released

New Features:

  • #147 TlmExtractor Full Column Names Mode
  • #148 TlmExtractor Share individual columns
  • #189 ScriptRunner Breakpoints don’t adapt to edits
  • #233 Add Config Option to Increase Tcpip Interface Timeout to TlmGrapher
  • #280 Method for determining interface packet count
  • #313 Add command line option to automatically start ScriptRunner
  • #336 Add Log Analyze Feature to TlmExtractor/CmdExtractor
  • #395 Implement Stylesheets Throughout
  • #408 Easy way to find which targets use an interface?
  • #433 Scripting support for TlmViewer close all screens
  • #434 TlmViewer option for no resize of screens
  • #436 PacketViewer option to ignore target.txt ignored items
  • #441 PacketViewer should identify derived items in the GUI

Maintenance:

None

Bug Fixes:

  • #417 Table Manager not checking ranges
  • #419 Support multiple arrays in string based commands

Migration Notes from COSMOS 3.8.x:

The Table Manager configuration file format has changed. Documentation will updated the first week of April.

You can migrate existing config files using:

bundle exec ruby tools\TableManager --convert config\tools\table_manager\old_table_def.txt

To upgrade to the latest version of COSMOS, run “bundle update cosmos” in your COSMOS project folder.

Packet Processors

COSMOS Packet Processors are a powerful concept that allow you to run code each time a specified packet is received. COSMOS provides a few generic Packet Processors which allows you to include statistics about individual telemetry points in your defined packets. Let’s break down how the COSMOS included processors are used and how you can implement your own Packet Processor.

First install COSMOS and start up the demo application. You’ll notice we declare a few targets of which one is called INST (for instrument). If you open up Packet Viewer and navigate to the INST target and the HEALTH_STATUS packet you can see a bunch of derived telemetry points at the top.

Packet Viewer

These points aren’t immediately obvious in the GUI (Ticket #441) but here they include all the items down to and including TEMP1STDDEV. If you right click on one of them and choose “Details” you can see that Data Type is DERVIED.

Derived

This is all controlled by the INST target’s cmd/tlm definition files. If you open the INST/cmd_tlm/inst_tlm.txt file from the demo you’ll see the following at the end of the HEALTH_STATUS packet definition:

  ITEM TEMP1HIGH 0 0 DERIVED "High-water mark for TEMP1"
    READ_CONVERSION processor_conversion.rb TEMP1WATER HIGH_WATER
  ITEM TEMP1LOW 0 0 DERIVED "Low-water mark for TEMP1"
    READ_CONVERSION processor_conversion.rb TEMP1WATER LOW_WATER
  ITEM TEMP1MAX 0 0 DERIVED "Maximum of most recent 100 samples for TEMP1"
    READ_CONVERSION processor_conversion.rb TEMP1STAT MAX
  ITEM TEMP1MIN 0 0 DERIVED "Minimum of most recent 100 samples for TEMP1"
    READ_CONVERSION processor_conversion.rb TEMP1STAT MIN
  ITEM TEMP1MEAN 0 0 DERIVED "Mean of most recent 100 samples for TEMP1"
    READ_CONVERSION processor_conversion.rb TEMP1STAT MEAN
  ITEM TEMP1STDDEV 0 0 DERIVED "Stddev of most recent 100 samples for TEMP1"
    READ_CONVERSION processor_conversion.rb TEMP1STAT STDDEV
  PROCESSOR TEMP1STAT statistics_processor.rb TEMP1 100
  PROCESSOR TEMP1WATER watermark_processor.rb TEMP1

These definitions create six new telemetry ITEMs. The READ_CONVERSION line takes a conversion class and then variable parameters that are passed to the class. Here we’re using the COSMOS provided processor_conversion.rb class which pulls a result calculated by a PROCESSOR. The last two lines define the two PROCESSORs.

Currently COSMOS provides the following three processors:

  1. new_packet_log_processor.rb - This processor creates a new packet log whenever the given Packet is seen.
  2. watermark_processor.rb - This processor monitors a telemetry item and tracks the high and low water points of that item since the launch of the Command and Telemetry Server.
  3. statistics_processor.rb - This processor collects a given number of samples of a telemetry item and calculates the minimum, maximum, mean, and standard deviation over the sample.

If all you want to do is to calculate useful statistics on your telemetry items you can stop reading now. For those who want to know how this works or want to implement their own Packet Processors, let’s continue into the source code.

Processor Implementation

require 'cosmos/processors/processor'
module Cosmos
  class WatermarkProcessor < Processor
    # @param item_name [String] The name of the item to gather statistics on
    # @param value_type #See Processor::initialize
    def initialize(item_name, value_type = :CONVERTED)
      super(value_type)
      @item_name = item_name.to_s.upcase
      reset()
    end

    # See Processor#call
    def call(packet, buffer)
      value = packet.read(@item_name, @value_type, buffer)
      high_water = @results[:HIGH_WATER]
      @results[:HIGH_WATER] = value if !high_water or value > high_water
      low_water = @results[:LOW_WATER]
      @results[:LOW_WATER] = value if !low_water or value < low_water
    end

    # Reset any state
    def reset
      @results[:HIGH_WATER] = nil
      @results[:LOW_WATER] = nil
    end

    # Convert to configuration file string
    def to_config
      "  PROCESSOR #{@name} #{self.class.name.to_s.class_name_to_filename} #{@item_name} #{@value_type}\n"
    end
  end
end

The initialize method gets passed the parameters from the config file. Thus our config file of: PROCESSOR TEMP1WATER watermark_processor.rb TEMP1
passes ‘TEMP1’ into ‘item_name’ of the initialize method: def initialize(item_name, value_type = :CONVERTED)
Since we only pass one value, we use the default value_type of :CONVERTED.

We store the item_name into a Ruby instance variable @item_name and call reset() to initialize our @results. But how did we get a @results instance variable? If you look at the class definition we are inheriting from Processor which is the base class for all COSMOS Processors. It declares a @results instance variable and initializes @results in its initialize method which we call using super(value_type).

The call method is the most important Processor method. It is always passed the packet and buffer. The packet is the COSMOS Packet instance which contains the value you’re interested in. Buffer is the raw binary buffer which this packet is based on. The Processor base class should never be directly used as it defines but does not implement call. Instead, you inherit from Processor like we did with WatermarkProcessor and implement your own call method. WatermarkProcessor reads the item we’re interested in and then compares it with the currently stored high and low value to determine if it should be saved. Note how it is saving the value in the @results hash with the :HIGH_WATER and :LOW_WATER symbol keys.

Processor Conversion

If you then open up the processor_conversion.rb code you can see how these results are converted into new telemetry items.

require 'cosmos/conversions/conversion'
module Cosmos
  # Retrieves the result from an item processor
  class ProcessorConversion < Conversion
    # @param processor_name [String] The name of the associated processor
    # @param result_name [String] The name of the associated result in the processor
    # @param converted_type [String or nil] The datatype of the result of the processor
    # @param converted_bit_size [Integer or nil] The bit size of the result of the processor
    def initialize(processor_name, result_name, converted_type = nil, converted_bit_size = nil)
      super()
      @processor_name = processor_name.to_s.upcase
      @result_name = result_name.to_s.upcase.intern
      if ConfigParser.handle_nil(converted_type)
        @converted_type = converted_type.to_s.upcase.intern
        raise ArgumentError, "Unknown converted type: #{converted_type}" if !BinaryAccessor::DATA_TYPES.include?(@converted_type)
      end
      @converted_bit_size = Integer(converted_bit_size) if ConfigParser.handle_nil(converted_bit_size)
    end

    # @param (see Conversion#call)
    # @return [Varies] The result of the associated processor
    def call(value, packet, buffer)
      packet.processors[@processor_name].results[@result_name] || 0 # Never return nil
    end
    def to_s; end # Not shown for brevity
    def to_config(read_or_write); end # Not shown for brevity
  end
end

First of all note that ProcessorConversion inherits from the Conversion base class. This is very similar to the WatermarkProcessor inheriting from the Processor base class. Again, there is an initialize method and a call method. The initialize method requires the processor_name and result_name and takes optional parameters that help describe the converted type. Let’s see how these map together in our definition.

Our config file looked like the following: READ_CONVERSION processor_conversion.rb TEMP1WATER HIGH_WATER
This passes TEMP1WATER and HIGH_WATER as processor_name and result_name into initialize: def initialize(processor_name, result_name, converted_type = nil, converted_bit_size = nil)

We store the processor name and result name into Ruby instance variables (first turning them into upper case strings). We additionally turn the result name into a Ruby symbol by calling intern on it. This allows us to match the symbol names we used in the WatermarkProcessor code.

All Conversion classes also implement the call method except with a slightly different signature. In addition to the packet and buffer being passed, the raw value is returned. The ProcessorConversion class uses the packet instance to access the processors hash by the given processor name and then accesses the results hash by the passed result name. We add a ‘|| 0’ which does a logical OR on the initial result to ensure that we don’t return a nil value as a result of the conversion.

Custom Processor

So how could we implement our own Processor? Let’s say you had some telemetry points that you wanted to average and report that averaged value as a new telemetry item. This is useful because you can then add limits to this new item and act on its value in scripts without having to constantly perform the averaging operation.

First create your new Processor class. Let’s call it MeanProcessor. This code should go into a file called mean_processor.rb and can either live in one of your target/lib folders or since it’s generic we can put it in the top level /lib directory in our project.

require 'cosmos/processors/processor'
module Cosmos
  class MeanProcessor < Processor
    # @param item_name [Array<String>] The names of the items to mean
    def initialize(*item_names) # the splat operator accepts a variable length argument list
      super(:CONVERTED) # Hard code to work on converted values
      @item_names = item_names # Array of the item names
      reset()
    end

    def call(packet, buffer)
      values = []
      @item_names.each do |item|
        values << packet.read(item, :CONVERTED, buffer)
      end
      @results[:MEAN] = values.inject(0, :+).to_f / values.length
    end

    # Reset any state
    def reset
      @results[:MEAN] = []
    end

    # Convert to configuration file string
    def to_config
      "  PROCESSOR #{@name} #{self.class.name.to_s.class_name_to_filename} #{@item_names.join(' ')}\n"
    end
  end
end

This class introduces some new Ruby syntax. Since we want to accept any number of items to average we have to accept a variable number of arguments in our initialize method. The ruby splat operator (or star operator) does this and places the arguments into a Ruby array. We store these names and then use them in our call method to perform the mean. I’m using a cool feature of Ruby’s Enumerable mixin, which is part of Array, to sum up the values (starting with 0) and then dividing by the number of values we have to get the mean. Note I’m also calling to_f to ensure the numerator is a floating point number so we do floating point math during the division. Integer division would truncate the value to an integer value.

First to use this new processor you need to require it in your target’s target.txt configuration file: REQUIRE mean_processor.rb
Then delcare the processing in your configuration definition as follows: TELEMETRY INST HEALTH_STATUS BIG_ENDIAN "Health and status from the instrument" ... # See demo configuration ITEM TEMPS_MEAN 0 0 DERIVED "Mean of TEMP1, TEMP2, TEMP3, TEMP4" READ_CONVERSION processor_conversion.rb TEMPMEAN MEAN PROCESSOR TEMPMEAN mean_processor.rb TEMP1 TEMP2 TEMP3 TEMP4

We define the processor on the INST HEALTH_STATUS packet and pass in 4 items to average. We also define a new derived item called TEMPS_MEAN which uses our previously described processor_conversion to pull out the MEAN value that we calculated. The result is shown in this PacketViewer screen shot:

Packet Viewer

Creating a custom processor definitely requires you to dive into the COSMOS API and play with the underlying Ruby code. Hopefully the existing processor code and this blog post helps you to derive whatever telemetry points you need. Happy processing and if you need additional support please contact us at cosmos@ball.com.

WIRED Article

Ball Aerospace COSMOS has been featured in an article on WIRED.com. The article focues on the fact that COSMOS is open source and what that means for the industry. The author includes quotes from GitHub’s VP of Product Engineering, Aerospace Corporation’s Principle Director of IT, and of course Ryan Melton and Jason Thomas of Ball Aerospace.

Ball Aerospace COSMOS 3.9.1 Released

New Features:

  • #382 CmdTlmServer Start/Stop for background tasks
  • #385 Quick access to COSMOS gem code
  • #388 Legal Dialog should show COSMOS version
  • #409 Update LINC interface to support multiple targets on the same interface

Maintenance:

  • #369 Table Manager refactoring
  • #386 Batch file for offline installation

Bug Fixes:

  • #236 Test Runner doesn’t support status_bar
  • #329 Using XTCE file instead of .txt cmd_tlm file didn’t work as online docs suggest
  • #378 TlmViewer displaying partials in the screen list
  • #402 Mac installation is failed - Please help.
  • #411 xtce explicit byte order list processing isn’t correct
  • #412 subscribe_packet_data needs to validate parameters

Migration Notes from COSMOS 3.8.x:

The Table Manager configuration file format has changed. Documentation will updated the first week of April.

You can migrate existing config files using:

bundle exec ruby tools\TableManager --convert config\tools\table_manager\old_table_def.txt

To upgrade to the latest version of COSMOS, run “bundle update cosmos” in your COSMOS project folder.

Routers

COSMOS Routers can be a confusing topic for people new and old alike to the COSMOS system. Let’s break them down and explore how COSMOS uses them internally and how you might use them in your configuration.

A COSMOS Router is first and foremost an interface like any other COSMOS interface. In fact if you look through the source code you won’t even see a Router class because they are instances of the Interface class. As a reminder, an Interface provides the lowest level connection between COSMOS and a target (something you’re trying to get COSMOS to talk to). Thus a Router is also providing a low level connection between COSMOS and something. So what’s the difference? Normal Interfaces read the target and send data (telemetry) to COSMOS and send commands to the target. Routers do the exact opposite: they route telemetry out to clients connected to them and route commands sent to them back to a target. Thus they provide a conduit for other clients to communicate with a target.

The COSMOS Command and Telemetry Server always starts a Router called the PREIDENTIFIED_ROUTER which appears in the Routers tab of the GUI: Server Routers

The Preidentified Router is controlled by the system.txt configuration file:

# Ethernet Ports
PORT CTS_API 7777
PORT TLMVIEWER_API 7778
PORT CTS_PREIDENTIFIED 7779

This Router is called the PREIDENTIFIED router because it uses a special stream protocol which adds the packet received time, the target name, and the packet name before sending the packet. This stream is used by the Telemetry Grapher so it doesn’t have to spend additional cycles identifying packets and can directly graph the packets it is interested in. The Replay tool also creates this router for the exact same purpose.

If you look back at the picture you’ll see in our demo configuration that we also create another Router called the INST_ROUTER. This is how that Router is configured:

ROUTER INST_ROUTER tcpip_server_interface.rb 2055 2055 10.0 nil LENGTH 32 16 7
  OPTION LISTEN_ADDRESS 127.0.0.1
  ROUTE INST_INT

Basically we’re creating a Router that routes only the INST interface data. We create this Router using a TCPIP Server interface with a LENGTH protocol of bit offset 32, length field size of 16, and length value offset of 7. These values were chosen to match the INST command and telemetry which use CCSDS headers. The CCSDS standard specifies a 16 bit length field at offset 32. The CCSDS standard also specifies that the length field is the length of the packet past the length field minus 1. Thus we add 7 for the 4 bytes of offset, 2 bytes of length field, and the minus 1.

For more information about the options you can pass to a Router (and Interface) see the Interface and Router Modifers part of the documentation.

To connect to the INST_ROUTER that we created we can write a little Ruby code using existing COSMOS interface classes and streams.

require 'cosmos'
require 'cosmos/interfaces/tcpip_client_interface'
i = Cosmos::TcpipClientInterface.new('localhost',2055,2055,nil,nil,'LENGTH',32,16,7)
i.connect
loop do
  pkt = Cosmos::System.telemetry.identify!(i.read.buffer, ['INST'])
  puts pkt.packet_name
end

Obviously we need the TCPIP Client Interface to match the port number of the TCPIP Server and we also need to match the LENGTH protocol options. Notice how we have to call the System.telemetry.identify! method to identify the raw packet stream. When I run this from a command prompt after starting the Server I see a stream of packet names similar to this:

MECH
ADCS
MECH
IMAGE
PARAMS
ADCS
HEALTH_STATUS
MECH
ADCS
...

You should see MECH and ADCS come out at 10Hz and the IMAGE, PARAMS, and HEALTH_STATUS come out at 1Hz. This matches the Packet Count ratio on the Tlm Packets tab of the Server.

We can also write some code to connect to the PREIDENTIFIED router. It looks very similar to the previous code except we do not need to identify the packets as they are pre-identified!

require 'cosmos'
require 'cosmos/interfaces/tcpip_client_interface'
i = Cosmos::TcpipClientInterface.new('localhost',7779,7779,nil,nil,'PREIDENTIFIED')
i.connect
loop do
  pkt = i.read
  puts pkt.packet_name
end

This time you should see some LIMITS_CHANGE packet names sprinkled in with the output from before. That’s because we’re accessing the ENTIRE COSMOS telemetry stream and not just the stream from the INST interface. Keep this in mind if you have performance issues when trying to process the entire telemetry stream.

Happy routing and if you need additional support please contact us at cosmos@ball.com.

Ball Aerospace COSMOS 3.8.3 Released

New Features:

  • #230 Make AUTO_TARGETS ignore target folders that have already been manually referenced
  • #257 Increment received_count in packet log reader
  • #264 Validate command/telemetry conversions during startup
  • #292 Target directory for API methods
  • #314 Update .bat files to handle spaces in path
  • #316 Add option to radiobutton widget to be checked by default
  • #326 Script Runner Crash using message_box with boolean parameter
  • #339 Add ‘Help’ menu item to open cosmosrb.com -> Documentation
  • #340 Packet Viewer - Allow select cell and copy value as text
  • #357 Add support for mixed endianness within tables
  • #359 Table Manager support MIN/MAX UINTX macros

Maintenance:

  • #349 Optimize cmd() to not build commands twice
  • #362 restore_defaults should take an optional parameter to exclude specified parameters
  • #365 Windows installer can have issues if .gem files are present in same folder
  • TestRunner support for newer Bundler (Abstract error when starting)

Bug Fixes:

  • #322 Udp interface thread does not gracefully shutdown
  • #327 TlmGrapher Screenshot in Linux captures the screenshot dialog box
  • #332 ERB template local variables dont’ support strings
  • #338 Setting received_time and received_count on a packet should clear the read conversion cache
  • #342 Cut and Paste Error in top_level.rb
  • #344 CmdTlmServer connect/disconnect button doesn’t work after calling connect_interface from script
  • #359 Table Manager doesn’t support strings
  • #372 TestRunner reinstantiating TestSuite/Test objects every execution

Migration Notes from COSMOS 3.7.x:

None

To upgrade to the latest version of COSMOS, run “bundle update cosmos” in your COSMOS project folder.

SmallSat Ops

The fact that COSMOS is the only open source C2 system has made it an ideal small satellite operations platform. This is one of the reasons Ryan attended the Small Satellite Conference back in August. He gave a very well received COSMOS presentation and was able to meet with many players in the industry.

Ryan at SmallSat Ryan enjoying the moment at Small Sat 2016

I recently googled for “COSMOS Command and Control” and came across a LinkedIn article about COSMOS. We had heard from Pat Stakem during the summer of 2015 but hadn’t heard much since. He and his students created a CubeSat using the open source Core Flight System from NASA Goddard. Of course they used COSMOS for their ground system running on Ubuntu. They integrated Apache to serve up the COSMOS telemetry files generated from Telemetry Extractor to operate their satellite “lights out”. They put together a great PowerPoint presentation with screenshots of COSMOS. They also created a whitepaper detailing the entire process.

Stakem Screenshot Screenshot showing COSMOS in action

Since COSMOS is open source, we don’t always hear about what people are doing with it. If you have a COSMOS success story you’d like to share with us please contact us at cosmos@ball.com.

ERB Config Files

COSMOS configuration files support ERB (Embedded RuBy) which is used heavily by Ruby on Rails. I found a pretty good description of ERB here. ERB allows you to put executable Ruby code in your configuration files. The trick is to surround the Ruby code with the special markers: <% <code> %>. If you want the result of your Ruby code to be placed in the configuration file you need to add the equal sign to the first marker: <%= <code> %>

A COSMOS user recently asked if he could include environment variables in his COSMOS configuration. This is very easy using the ERB syntax. For example, you have an environment variable named “LOG_DIR” in your system which points to the path you want to store your COSMOS logs. To use this value you would modify your system.txt file as follows:

...
# Paths
PATH LOGS <%= ENV["LOG_DIR"] %>
...

When this file gets parsed by COSMOS, the value of the LOG_DIR environment variable gets inserted into the system.txt output file. Note the %= syntax to insert the value and how I’m using the ENV from the Ruby core library.

It’s recommended that you don’t put too much logic in these ERB statements to keep your configuration files readable and maintainable. If you have a complex piece of code you want to use in an ERB statement, you can create a utility in your ‘lib’ folder and define methods to use. For example, in your ‘lib’ folder create utilties.rb:

def log_path
  File.join(Cosmos::USERPATH, 'outputs', 'mylogs')
end

Now in system.txt we can use that ‘log_path’ routine after we first require ‘utilities’.

<% require 'utilities' %>
...
# Paths
PATH LOGS <%= log_path() %>
...

Notice how the first ERB statement does NOT use the %= syntax since I’m simply requiring the file I want to use. I don’t want to put anything in the template itself. Later in the PATH statement I use the %= syntax to insert the result of the log_path() method.

ERB templates are particularly useful in command and telemetry definitions as they allow you to reuse sections. We’ve added our own routine called ‘render’ (similar to Ruby on Rails) which can render a command or telemetry template. The best example of this is in the COSMOS Demo INST target. If you open the inst_cmds.txt file you’ll see this:

COMMAND INST COLLECT BIG_ENDIAN "Starts a collect on the instrument"
  <%= render "_ccsds_cmd.txt", locals: {id: 1} %>
  PARAMETER    TYPE           64  16  UINT MIN MAX 0 "Collect type"
    REQUIRED
    STATE NORMAL  0
    STATE SPECIAL 1 HAZARDOUS
  PARAMETER    DURATION       80  32  FLOAT 0.0 10.0 1.0 "Collect duration"
  PARAMETER    OPCODE        112   8  UINT 0x0 0xFF 0xAB "Collect opcode"
    FORMAT_STRING "0x%0X"
  PARAMETER    TEMP          120  32  FLOAT 0.0 25.0 0.0 "Collect temperature"
    UNITS Celcius C

COMMAND INST ABORT BIG_ENDIAN "Aborts a collect on the instrument"
  <%= render "_ccsds_cmd.txt", locals: {id: 2} %>

...

Notice the call to <%= render "_ccsds_cmd.txt", locals: {id: 1} %>. Opening the ‘_ccsds_cmd.txt’ file reveals this command template:

  PARAMETER    CCSDSVER        0   3  UINT  0     0   0 "CCSDS primary header version number"
  PARAMETER    CCSDSTYPE       3   1  UINT  1     1   1 "CCSDS primary header packet type"
  PARAMETER    CCSDSSHF        4   1  UINT  0     0   0 "CCSDS primary header secondary header flag"
  ID_PARAMETER CCSDSAPID       5  11  UINT  0  2047 999 "CCSDS primary header application id"
  PARAMETER    CCSDSSEQFLAGS  16   2  UINT  3     3   3 "CCSDS primary header sequence flags"
  PARAMETER    CCSDSSEQCNT    18  14  UINT  0 16383   0 "CCSDS primary header sequence count"
    OVERFLOW TRUNCATE
  PARAMETER    CCSDSLENGTH    32  16  UINT MIN MAX 12 "CCSDS primary header packet length"
  ID_PARAMETER PKTID          48  16  UINT MIN MAX <%= id %> "Packet id"

The call to render replaces everything in the named template with the render call. We follow the Ruby on Rails convention of naming these templates (Rails calls them ‘partials’) with a leading underscore to differentiate them from full command and telemetry definitions. Notice too that we are passing local variables to the template. The ‘id: 1’ syntax is basically setting the ‘id’ variable in the template to 1. This allows us to send a different PKTID to each command.

ERB is incredibly powerful and a great way to avoid WET (Write Each Time) command and telemetry definitions. Now go DRY (Don’t Repeat Yourself) up your COSMOS configuration and as always if you need additional support please contact us at cosmos@ball.com.

Custom Widgets

Sometimes we receive requests to make custom COSMOS widgets or to modify existing COSMOS widgets to add certain looks or functionality. While this is a project we’re happy to perform for our customers, it’s also something that can be done by end users willing to dig into some of the Qt and COSMOS documentation. In this post, I’m going to describe how to create a custom COSMOS widget.

When asked to perform customizations like this I first bring up the COSMOS Demo. We try to include all the COSMOS features in the Demo so end users have concrete examples to follow instead of relying solely on the excellent documentation at cosmosrb.com. Obviously you must first have COSMOS installed so follow the installation instructions and then launch the Demo by running the Launcher in the Demo folder. Here is how the server appears on my Windows machine:

COSMOS Demo Server

I’m going to create a custom widget in the INST target to display some of the Array data in a table. If you first launch the Telemetry Viewer and open the INST ARRAY screen you should see the following:

COSMOS Inst Array

This screen is already using the Array widget to display this data in a text box. We will add our new widget to the top of the screen which will display the data in a table. Let’s add the line to the screen which will call our new widget. Edit demo/config/targets/INST/screens/array.txt and add the following line in the middle:

...
TITLE "Instrument Array Data"
DEMOTABLE INST HEALTH_STATUS ARY
ARRAY INST HEALTH_STATUS ARY 300 50 nil 8 FORMATTED
...

Now we need to create the DemotableWidget which will implement the actual display. Create a new file called demotable_widget.rb in demo/config/targets/INST/lib. Note that the name of the line in the config file, DEMOTABLE, must be all lowercase followed by an underscore and ‘widget’. The class name in the file must be one word with the first letter and Widget capitalized. This is how it should start:

require 'cosmos/tools/tlm_viewer/widgets/widget'

module Cosmos
  class DemotableWidget < Qt::TableWidget
    include Widget

    def initialize(parent_layout, target_name, packet_name, item_name, value_type = :WITH_UNITS)
      super(target_name, packet_name, item_name, value_type)
    end
  end
end

We’re extending the closest widget that Qt offers to what we’re trying to achieve. In this case it’s pretty obvious but you can get documentation on all the Qt classes. In many cases it might be easier to extend an existing COSMOS widget.

Note that our initialize method takes the parent_layout as the first value. All COSMOS widgets make the first parameter the parent_layout so they can be added. The next four paramaters are typically the target_name, packet_name, item_name and value_type. Additional parameters can follow the value_type parameter. The first thing we do in the initialize method is call super which calls the Widget initialize method. If you run this code you should see that the screen displays but doesn’t look any different. That’s because we haven’t actually added our new widget to the parent_layout. Before adding widgets to the layout you typically want to configure them. For our table, we need to set the number of rows and columns. First I grab the telemetry value from the server using the System.telemetry.value method defined in telemetry.rb. Since this is an array value I call length to determine how many rows to display in the table. I then use the Qt methods setRowCount and setColumnCount to initialize the table. You can find these methods in the Qt::TableWidget documentation. Finally I call the addWidget method which is a part of all the Qt::Layout classes.

    def initialize(parent_layout, target_name, packet_name, item_name, value_type = :WITH_UNITS)
      super(target_name, packet_name, item_name, value_type)
      value = System.telemetry.value(target_name, packet_name, item_name) # Get the value
      @rows = value.length # Store the rows
      setRowCount(@rows)
      setColumnCount(1)
      parent_layout.addWidget(self) if parent_layout
    end

Now if you stop and restart the Telemetry Viewer (so it can re-require the new widget code) it should display an empty table:

COSMOS Inst Array

To actually populate it with data we must follow the Cosmos Widget conventions. First of all by including Widget you include all the Widget code which creates two key class methods: layout_manager? and takes_value?. These must be overridden to return true if your widget is either a layout or takes a value respectively. Since our widget will be taking the array data as a value we must override takes_value?:

require 'cosmos/tools/tlm_viewer/widgets/widget'

module Cosmos
  class DemotableWidget < Qt::TableWidget
    include Widget

    def self.takes_value?
      return true
    end

Typically class methods are defined at the top of the source file and begin with self. You can also type out the class name but this is less robust as changing the class name requires changing the method name. Implementing this class method allows Telemetry Viewer to call the value=(data) method with new telemetry data. The value method implementation should look like this:

    def value=(data)
      (0...@rows).each do |row| # Note the extra 'dot' which means up to but not including
        setItem(row, 0, Qt::TableWidgetItem.new(data[row].to_s))
      end
    end

The data value passed to the method is the same target, packet, and item used in the screen definition. In our value= method we are using our stored instance variable @rows to index into the array data and create new Qt::TableWidgetItem instances to store the data. TableWidgetItems expect Strings to be passed so I call to_s on the data item to ensure it is a String. If you now re-launch Telemetry Viewer you should see the values populated in the table:

COSMOS Inst Array

At this point you could be done. But wait! The Array widget below the table fades darker to implement “aging”, showing the user the values haven’t changed. How do we implement “aging” in our new widget? To start we require the aging_widget and include the AgingWidget module. Then we must call the setup_aging method in our initialize method as well as redefine the process_settings method:

require 'cosmos/tools/tlm_viewer/widgets/widget'
require 'cosmos/tools/tlm_viewer/widgets/aging_widget'

module Cosmos
  class DemotableWidget < Qt::TableWidget
    include Widget
    include AgingWidget

    def initialize(parent_layout, target_name, packet_name, item_name, value_type = :WITH_UNITS)
      super(target_name, packet_name, item_name, value_type)
      setup_aging()
      value = System.telemetry.value(target_name, packet_name, item_name) # Get the value
      @rows = value.length # Store the rows
      setRowCount(@rows)
      setColumnCount(1)
      parent_layout.addWidget(self) if parent_layout
    end

    def process_settings
      super
      process_aging_settings
    end
  end
end

Note that we were able to remove the class method self.takes_value? because AgingWidget already implements it. This is all required to setup aging but we must still modify the value= method to do the work. First in value= we call super to call the AgingWidget’s value= method. This method returns a string representation of the data with the correct foreground color and text character indicating the color, e.g. G=Green, Y=Yellow, R=Red. This is important for values with limits settings but since our array value doesn’t have limits I’m going to igore the return value and simply allow the aging routine to age the data. Interally this updates the @background instance variable with the current ‘aged’ background color. I then set the TableWidgetItem’s background color to this color before adding it to the table:

    def value=(data)
      super(data)
      (0...@rows).each do |row|
        item = Qt::TableWidgetItem.new(data[row])
        item.setBackgroundColor(@background)
        setItem(row, 0, item)
      end
    end

The end result is aging:

COSMOS Inst Array

Note that if you have a widget that implements aging and limits you’ll want to keep the value returned by super and use it in your widget. If you don’t want the aging routine to directly use your data value you can pass a string as the second parameter, e.g. super(data, text). This text string will be modified with the color blind settings. Basically that means that whatever the calculated @foreground color string is, a corresponding text character is added (R=Red, G=Green, etc) to aid people who can’t distinguish colors. See aging_widget.rb for more details.

Good luck creating your own widgets and if you need additional support please contact us at cosmos@ball.com.

Custom COSMOS Interface

One of our Ball Aerospace engineers asked how they could add a checksum to an existing COSMOS interface when talking to their target. COSMOS does not support this directly so it requires creating a custom interface. While this might sound daunting, the COSMOS interfaces were designed just for this type of extension and provide hooks for customization.

In this example we will assume the original interface is the COSMOS Serial Interface. In your target’s lib folder create a new interface called checksum_serial_interface.rb:

require 'cosmos' # always require cosmos
require 'cosmos/interfaces/serial_interface' # original interface being extended

module Cosmos
class ChecksumSerialInterface < SerialInterface
  def pre_write_packet(packet)
    data = packet.buffer
    checksum = 0xFFFF
    data.each_byte {|x| checksum += x }
    checksum &= 0xFFFF
    data << [checksum].pack("n") # Pack as 16 bit unsigned bit endian
    return data
  end

  def post_read_data(packet_data)
    len = packet_data.length
    calc_checksum = 0xFFFF
    packet_data[0..(len - 3)].each_byte {|x| calc_checksum += x }
    calc_checksum &= 0xFFFF
    rx_checksum = packet_data[-2..-1].unpack("n") # Unpack as 16 bit unsigned big endian
    if calc_checksum == rx_checksum
      return packet_data
    else
      puts "Bad checksum detected. Calculated: 0x#{calc_checksum.to_s(16)} Received: 0x#{rx_checksum.to_s(16)}. Dropping packet."
      return "" # Also can return nil to break the connection and reconnect to the target
    end
  end

end
end

What we’re doing is overriding pre_write_packet in StreamInterface to allow us to modify the data before it is written to the packet and sent over the interface. We also override post_read_data to operate on data received before it is sent back to the COSMOS server and thus the tools. Note there is also a post_read_packet(packet) method which is called after post_read_data is called and after the COSMOS Packet has been created. All Interfaces inheriting from StreamInterface includes these callback methods, including SerialInterface, TcpipServerInterface, and TcpipClientInterface. Note that UdpInterface inherits directly from Interface and thus does NOT include these callbacks.

Then in your cmd_tlm_server.txt file for your target you use your new interface:

#         interface name  file name                    write read baud   parity stop timeouts stream
INTERFACE UART_INTERFACE  checksum_serial_interface.rb COM1  COM1 115200 NONE   1    nil nil  BURST

I added a comment line above the definition which describes the settings. For more information see the Serial Interface documentation.

This same technique can obviously be used to extend the the other TCPIP interfaces and can be used with all the various Stream Protocol classes COSMOS defines.

COSMOS Cmd/Tlm Naming

Recently a user asked if they could add exclamation points and question marks to their command and telemetry items. Absolutely! COSMOS provides great flexibility in command and telemetry naming conventions. (See cmdtlm). For example, adding an exclamation point to a command to denote a more severe version of the same command:

COMMAND TGT ABORT BIG_ENDIAN "Tries to abort a collect on the instrument"
COMMAND TGT ABORT! BIG_ENDIAN "Force aborts a collect on the instrument"

While it doesn’t make sense to define a command with a question mark, it works well with telemetry points. For example, there is a telemetry point which indicates whether a mechanism is deployed or not. It is an analog value that indicates deployed if the value is above zero. To view both the raw value and the deployed status, define a derived telemetry point which indicates a TRUE or FALSE status:

APPEND_ITEM DEPLOYED 16 UINT "Deployed raw value"
ITEM DEPLOYED? 0 0 DERIVED "Deployed status"
  STATE FALSE 0
  STATE TRUE 1
  GENERIC_READ_CONVERSION_START UINT 8
    myself.read('DEPLOYED') > 0 ? 1 : 0
  GENERIC_READ_CONVERSION_END

Note that this is probably overkill in this case because the conversion could just as easily be applied directly to the item. The raw value could then be obtained by calling tlm_raw(“TGT PKT DEPLOYED”) (see tlm_raw).

These practices are similar to the Ruby convention of using methods with an exclamation point (bang) to indicate a dangerous method which typically directly modifies its caller. Ruby also has a convention of methods with question marks returning a boolean true or false value. Read more in the Ruby documentation.

Ball Aerospace COSMOS 3.8.2 Released

New Features:

Maintenance:

  • COSMOS Downloads graph rake task updates

Bug Fixes:

  • #303 Need to clear read conversion cache on Packet#clone
  • #304 Win32 Serial Driver clean disconnect
  • #309 Fix Script Runner insert_return when not running

Migration Notes from COSMOS 3.7.x:

None

Ball Aerospace COSMOS 3.8.1 Released

New Features:

  • #184 Limits Monitor show green for blue limits items
  • #190 Simpler MIN MAX syntax for command definitions
  • #254 Get buffer from commands
  • #259 Proper support for user selected text editor on linux
  • #262 PackerViewer option for listing derived items last
  • #271 Time.formatted option for no microseconds
  • #288 check_tolerance should enforce a positive tolerance
  • #301 Update use of COSMOS_DEVEL

Maintenance:

  • #268 xtce_converter doesn’t support byte order list
  • #277 Test Runner support for Script Runner options
  • #285 xtce converter doesn’t support LocationInContainerInBits

Bug Fixes:

  • #256 Defining initialize method in Cosmos::Test class breaks the class when using Test Selection in TestRunner
  • #273 Wrap Qt::Application.instance in main_thread
  • #287 Installer issue on newer versions of Ubuntu and Debian related to libssl
  • #293 Units applied after a read_conversion that returns a string modifies cached conversion value
  • #294 String#convert_to_value should always just return the starting string if the conversion fails
  • #298 COSMOS IoMultiplexer breaks gems that invoke stream operator on STDOUT/ERR

Migration Notes from COSMOS 3.7.x:

None

Sparkfun Blog Post

Ball Aerospace COSMOS has been featured in a blog post on SparkFun. SparkFun is an online retail store that sells electronics primarily for hobbyists. Ball recently purchased a Raspberry Pi, Arduino, and various other parts to interface to COSMOS. We think hobbyists will find COSMOS a great way to interface to their embedded projects and SparkFun agrees! Get started with COSMOS now!

XTCE Support

Ball Aerospace COSMOS now has support for the XTCE Command and Telemetry Definition Standard. This is an open standard designed to allow command and telemetry definitions to be transferred between different ground systems. COSMOS can run directly using the .xtce files, or can convert them into the COSMOS configuration file format.

See the docs for more information: XTCE Support

Fukuoka Special Award Winner

Ball Aerospace COSMOS has been honored by the Ruby community by receiving one of the three Fukuoka Special Awards handed out this year! This is a great honor with the contest judging being led by Matz himself (the creator of the Ruby programming language). Read more on the MyFukuoka website here: MyFukuoka 2016 Ruby Award Winners

Ball Aerospace COSMOS 3.8.0 Released

With this release COSMOS now has initial support for the XTCE Command and Telemetry Definition standard.

New Features:

  • #251 Create COSMOS XTCE Converter
  • #252 Add polling rate command line option to PacketViewer

Bug Fixes:

  • #245 TlmGrapher Crashes on Inf
  • #248 Can’t script commands containing ‘with’ in the name

Migration Notes from COSMOS 3.7.x:

None

COSMOS Simulated Target

Creating a COSMOS Simulated Target

Sometimes you have a need to create a simulated target in COSMOS. This simulated target is not a physical target producing data and accepting commands but a software target which generates data and sends it to COSMOS. This is exactly how the COSMOS Demo operates within the INST target that it creates. While this is a very full featured example its complexity can be a little overwhelming. In this post I’m going to break down a much simpler simulated target so you can create your own.

First of all create a new COSMOS target directory in config/targets. I called mine INST (instrument) to match the COSMOS demo. Create the ‘cmd_tlm’ and ‘lib’ subdirectories. For my demo I created a simple ‘cmd.txt’ file which contains a single command:

COMMAND INST SET_STATUS BIG_ENDIAN "Set status"
  APPEND_PARAMETER STATUS 0 STRING "STATUS" "Status"
    STATE "OK" "OK"
    STATE "ERROR" "ERROR"

I created a ‘tlm.txt’ file which contains two different telemetry packets:

TELEMETRY INST STATUS BIG_ENDIAN "Status from the instrument"
  APPEND_ID_ITEM ID 16 UINT 1 "Packet ID"
  APPEND_ITEM COUNTER 16 UINT "Packet counter"
  APPEND_ITEM STATUS 0 STRING "Most recent ASCIICMD string"
    STATE "OK" "OK"
    STATE "ERROR" "ERROR"

TELEMETRY INST DATA BIG_ENDIAN "Data from the instrument"
  APPEND_ID_ITEM ID 16 UINT 2 "Packet ID"
  APPEND_ITEM COUNTER 16 UINT "Packet counter"
  APPEND_ITEM TIMESEC 32 UINT "Seconds since epoch (January 1st, 1970, midnight)"
  APPEND_ITEM TIMEUS  32 UINT "Microseconds of second"
  APPEND_ITEM TEMP1 32 INT "Temperature #1"
    UNITS CELCIUS C
    FORMAT_STRING "%0.3f"
    LIMITS DEFAULT 1 ENABLED -80.0 -70.0 60.0 80.0 -20.0 20.0

The cmd_tlm_server.txt file is very simple:

INTERFACE INST_INT simulated_target_interface.rb sim_inst.rb
  TARGET INST

The real work is in implementing how your simulated target is going to behave. This is done in the lib/sim_inst.rb file. Note that whatever you name your simulated target file must match the last parameter of the INTERFACE in the cmd_tlm_server.rb as shown above.

I’ll break down my sim_inst.rb piece by piece and then list it in its entirety. First you must inherit from the Cosmos::SimulatedTarget.

require 'cosmos'
module Cosmos
  class SimInst < SimulatedTarget

Next you can initialize any of your packets in the initialize method. This is entirely optional but I show how to use the @tlm_packets hash to access all the defined packets. This hash is created automatically by the SimulatedTarget based on all the packets you have defined in your cmd_tlm/tlm.txt file. Note that there is NOT a corresponding @cmd_packets.

def initialize(target_name)
  super(target_name)

  # We grab the STATUS packet to set initial values
  packet = @tlm_packets['STATUS']
  packet.enable_method_missing # required to use packet.<item> = value
  packet.status = "NONE"
end

We then have to configure the telemetry packet rates of our target. That is, how fast do the packets get sent out. This is handled by implementing the set_rates method and by calling set_rate for each packet defined in your system. If you do not call set_rate the packet will not be send out periodically (which may be desirable for event based packets).

def set_rates
  # The SimulatedTarget operates on a 101Hz clock
  # Thus the rates are determined by dividing this rate
  # by the set rate to get the output rate of the packet
  set_rate('STATUS', 100) # 100 / 100 = 1Hz
  set_rate('DATA', 10) # 100 / 10 = 10Hz
end

If your target will accept command you need to implemented the write(packet) method. My write method is simple in that I only have a single command that directly sets a value in one of my telemetry packets.

def write(packet)
  # We directly set the telemetry value from the only command
  # If you have more than one command you'll need to switch
  # on the packet.packet_name to determine what command it is
  @tlm_packets['STATUS'].status = packet.read("status")
end

Your target must implement the read(count_100hz, time) method to return telemetry packets back to COSMOS. You’ll call the get_pending_packets(count_100hz) method implemented by SimulatedTarget and then perform whatever operations you want on the packets before returning the array of packets back to COSMOS. Note my use of the cycle_tlm_item method to automatically cycle the telemetry item as each packet is sent out. This is used heavily in the COSMOS Demo.

def read(count_100hz, time)
  # The SimulatedTarget implements get_pending_packets to return
  # packets at the correct time interval based on their rates
  pending_packets = get_pending_packets(count_100hz)

  pending_packets.each do |packet|
    case packet.packet_name
    when 'STATUS'
      packet.counter += 1
    when 'DATA'
      # This method in SimulatedTarget cycles the specified telemetry
      # point between the two given values by the given increment for
      # each packet sent out.
      cycle_tlm_item(packet, 'temp1', -95.0, 95.0, 1.0)

      packet.timesec = time.tv_sec
      packet.timeus  = time.tv_usec
      packet.counter += 1
    end
  end
  pending_packets
end

Hopefully that a little easier to understand than the full COSMOS Demo which has much more complex command and telemetry definitions and simulated targets in order to better exercise the various COSMOS tools. While there are other ways to simulate COSMOS targets they can get you into trouble if you’re not careful about properly cloning packets sending back updated data. Additionally, using the SimulatedTargetInterface in your Interface makes it very clear to other developers that this target is indeed simulated.

Without further ado, here is my sim_inst.rb in its entirety:

require 'cosmos'
module Cosmos
  class SimInst < SimulatedTarget
    def initialize(target_name)
      super(target_name)

      # We grab the STATUS packet to set initial values
      packet = @tlm_packets['STATUS']
      packet.enable_method_missing # required to use packet.<item> = value
      packet.status = "NONE"
    end

    def set_rates
      # The SimulatedTarget operates on a 100Hz clock
      # Thus the rates are determined by dividing this rate
      # by the set rate to get the output rate of the packet
      set_rate('STATUS', 100) # 100 / 100 = 1Hz
      set_rate('DATA', 10) # 100 / 10 = 10Hz
    end

    def write(packet)
      # We directly set the telemetry value from the only command
      # If you have more than one command you'll need to switch
      # on the packet.packet_name to determine what command it is
      @tlm_packets['STATUS'].status = packet.read("status")
    end

    def read(count_100hz, time)
      # The SimulatedTarget implements get_pending_packets to return
      # packets at the correct time interval based on their rates
      pending_packets = get_pending_packets(count_100hz)

      pending_packets.each do |packet|
        case packet.packet_name
        when 'STATUS'
          packet.counter += 1
        when 'DATA'
          # This method in SimulatedTarget cycles the specified telemetry
          # point between the two given values by the given increment for
          # each packet sent out.
          cycle_tlm_item(packet, 'temp1', -95.0, 95.0, 1.0)

          packet.timesec = time.tv_sec
          packet.timeus  = time.tv_usec
          packet.counter += 1
        end
      end
      pending_packets
    end
  end
end

Happy simulated target programming!

Ball Aerospace COSMOS 3.7.1 Released

Bug Fixes:

  • #228 Fix typo in udp_interface
  • #231 MACRO_APPEND with multiple items not working
  • #235 Improve IntegerChooser and FloatChooser Validation
  • #236 TestRunner doesn’t support status_bar
  • #240 Make sure super() is called in all bundled conversion classes
  • #241 Don’t reformat BLOCK data types with a conversion in Structure#formatted

Migration Notes from COSMOS 3.6.x:

  1. Background task arguments are now broken out instead of being received as a single array
  2. udp_interface now takes an optional argument for bind_address
  3. line_graph_script has been significantly updated to support modifying plots from the script.

Ball Aerospace COSMOS 3.7.0 Released

New Features:

  • #213 Vertical Limits Bar
  • #214 TlmGrapher show full date for plotted points
  • #219 State Color Widget
  • #225 Set Bind Address in UDP interface

Maintenance:

  • #223 C Extension Improvements

Bug Fixes:

  • #199 Investigate TlmGrapher Formatted Time Item
  • #211 Background task with arguments not working
  • #217 Graph Right Margin Too Small

Migration Notes from COSMOS 3.6.x:

  1. Background task arguments are now broken out instead of being received as a single array
  2. udp_interface now takes an optional argument for bind_address
  3. line_graph_script has been significantly updated to support modifying plots from the script.

Ball Aerospace COSMOS 3.6.3 Released

New Features:

  • #200 ScriptRunner Find Dialog Does Not Cross Windows
  • #201 Table Manager to support arbitrary inputs on State Fields
  • #209 Add UTS Timestamp Flag to TlmGrapher Plots

Maintenance:

  • #194 Allow up to one minute for TlmViewer to start when calling display() from a script
  • #203 load_utility should raise LoadError like load and require
  • #205 Add testing for array and matrix

Bug Fixes:

  • #191 Installing COSMOS Issue on Windows 7
  • #193 Fix ask() on linux and qt 4.6.2
  • #197 Improve linc interface

Migration Notes from COSMOS 3.5.x:

None

Ball Aerospace COSMOS 3.6.2 Released

Huge new feature in this release: All COSMOS configuration files are now interpreted with the ERB preprocessor! This allows you to use Ruby code within the configuration files to help build them. You can also render partials of common information such as packet headers so you only have to define them once. See the INST target in the updated Demo project for examples.

Bug Fixes:

  • #187 Must require tempfile in config_parser.rb on non-windows systems

Migration Notes from COSMOS 3.5.x:

None

Ball Aerospace COSMOS 3.6.0 Released

Huge new feature in this release: All COSMOS configuration files are now interpreted with the ERB preprocessor! This allows you to use Ruby code within the configuration files to help build them. You can also render partials of common information such as packet headers so you only have to define them once. See the INST target in the updated Demo project for examples.

Bug Fixes:

  • #168 Select unreliably unblocks when closing sockets on linux
  • #177 MACRO_APPEND in descending order is broken
  • #179 ScriptRunnerFrame Context Menu Crash
  • #182 Overriding LOG_WRITERS in cmd_tlm_server.txt can cause issues

New Features:

  • #170 Consider supporting a preprocessor over COSMOS config files
  • #171 Script Runner should have file open and save GUI dialogs
  • #174 Add View in Command Sender in Server

Maintenance:

  • #80 Investigate performance of nonblocking IO without exceptions

Migration Notes from COSMOS 3.5.x:

None

Ball Aerospace COSMOS 3.5.3 Released

Bug Fixes:

  • #169 Make windows bat files support running outside of the current directory

New Features:

  • N/A

Maintenance:

  • N/A

Migration Notes from COSMOS 3.4.2:

The launcher scripts and .bat files that live in the COSMOS project tools folder have been updated to be easier to maintain and to ensure that the user always sees some sort of error message if a problem occurs starting a tool. All users should copy the new files from the tools folder in the COSMOS demo folder into their projects as part of the upgrade to COSMOS 3.5.1

COSMOS now disables reverse DNS lookups by default because they can take a long time in some environments. If you still want to see hostnames when someone connects to a TCP/IP server interface/router then you will need to add ENABLE_DNS to your system.txt file.

Ball Aerospace COSMOS 3.5.2 Released

Bug Fixes:

  • #167 Use updated url for wkhtmltopdf downloads

New Features:

  • #166 Add install script for Ubuntu

Maintenance:

  • N/A

Migration Notes from COSMOS 3.4.2:

The launcher scripts and .bat files that live in the COSMOS project tools folder have been updated to be easier to maintain and to ensure that the user always sees some sort of error message if a problem occurs starting a tool. All users should copy the new files from the tools folder in the COSMOS demo folder into their projects as part of the upgrade to COSMOS 3.5.1

COSMOS now disables reverse DNS lookups by default because they can take a long time in some environments. If you still want to see hostnames when someone connects to a TCP/IP server interface/router then you will need to add ENABLE_DNS to your system.txt file.

Ball Aerospace COSMOS 3.5.1 Released

This release fixes a bug and completes the installation scripts for linux/mac.

Bug Fixes:

  • #165 Change launch_tool to tool_launch in Launcher

New Features:

  • N/A

Maintenance:

  • #102 Create Installation Scripts

Migration Notes from COSMOS 3.4.2:

The launcher scripts and .bat files that live in the COSMOS project tools folder have been updated to be easier to maintain and to ensure that the user always sees some sort of error message if a problem occurs starting a tool. All users should copy the new files from the tools folder in the COSMOS demo folder into their projects as part of the upgrade to COSMOS 3.5.1

COSMOS now disables reverse DNS lookups by default because they can take a long time in some environments. If you still want to see hostnames when someone connects to a TCP/IP server interface/router then you will need to add ENABLE_DNS to your system.txt file.

Ball Aerospace COSMOS 3.5.0 Released

This release contains a lot of new functionality and a key new feature: The ability to create new COSMOS targets and tools as reusable gems! This will hopefully allow the open source community to create sharable configuration for a large amount of hardware and allow for community generated tools to be easily integrated.

Bug Fixes:

  • #153 set_tlm should support settings strings with spaces using the normal syntax
  • #155 Default to not performing DNS lookups

New Features:

  • #25 Warn users if reading a packet log uses the latest instead of the version specified in the file header
  • #106 Allow the server to run headless
  • #109 Cmd value api
  • #129 Script Runner doesn’t syntax highlight module namespacing
  • #133 Add sound to COSMOS alerts
  • #138 Limits Monitor should show what is stale
  • #142 Support gem based targets and tools
  • #144 Never have nothing happen when trying to launch a tool
  • #152 Provide a method to retrieve current suite/group/case in TestRunner
  • #157 Launcher support command line options in combobox
  • #163 Allow message_box to display buttons vertically

Maintenance:

  • #131 Consolidate Find/Replace logic in the FindReplaceDialog
  • #137 Improve Server message log performance
  • #142 Improve Windows Installer bat file
  • #146 Need support for additional non-standard serial baud rates
  • #150 Improve Win32 serial driver performance

Migration Notes from COSMOS 3.4.2:

The launcher scripts and .bat files that live in the COSMOS project tools folder have been updated to be easier to maintain and to ensure that the user always sees some sort of error message if a problem occurs starting a tool. All users should copy the new files from the tools folder in the COSMOS demo folder into their projects as part of the upgrade to COSMOS 3.5.0

COSMOS now disables reverse DNS lookups by default because they can take a long time in some environments. If you still want to see hostnames when someone connects to a TCP/IP server interface/router then you will need to add ENABLE_DNS to your system.txt file.

Ball Aerospace COSMOS 3.4.2 Released

Issues:

  • #123 TestRunner command line option to launch a test automatically
  • #125 Fix COSMOS issues for qtbindings 4.8.6.2
  • #126 COSMOS GUI Chooser updates

Migration Notes from COSMOS 3.3.x or 3.4.x:

COSMOS 3.4.2 requires qtbindings 4.8.6.2. You must also update qtbindings when installing this release. Also note that earlier versions of COSMOS will not work with qtbindings 4.8.6.2. All users are strongly recommended to update both gems.

Ball Aerospace COSMOS 3.4.1 Released

Issues:

  • #121 BinaryAccessor write crashes with negative bit sizes

Migration Notes from COSMOS 3.3.x:

None

Note: COSMOS 3.4.0 has a serious regression when writing to variably sized packets. Please upgrade to 3.4.1 immediately if you are using 3.4.0.

Ball Aerospace COSMOS 3.4.0 Released

Issues:

  • #23 Handbook Creator User’s Guide Mode
  • #72 Refactor binary_accessor
  • #101 Support Ruby 2.2 and 64-bit Ruby on Windows
  • #104 CmdTlmServer Loading Tmp & SVN Conflict Files
  • #107 Remove truthy and falsey from specs
  • #110 Optimize TlmGrapher
  • #111 Protect Interface Thread Stop from AutoReconnect
  • #114 Refactor Cosmos::Script module
  • #118 Allow PacketViewer to hide ignored items

Migration Notes from COSMOS 3.3.x:

None

COSMOS BinaryAccessor#write C Extension

How I created a COSMOS C Extension

The COSMOS framework has several C extensions created to increase performance. One critical piece that was created early on is the extension to the BinaryAccessor class. This allows for increased performance when reading items from binary packets which is the most common operation in the COSMOS system. I created a COSMOS performance configuration which spawns 30 fake targets and attempts to send commands to them as fast as possible. Sending commands exercies the write portion of BinaryAccessor and profiling showed this was now becoming a bottleneck. Therefore I set out to port the write method to the existing C extension.

The fact that Ryan had already implemented the read method as a C extension gave me a huge head start. I first copied all the Ruby code directly into the C extension so I could try to translate it line by line. Initially if I didn’t know how to do the translation I would just comment it out and see how much I could compile. A nice way to do this in C code is use #if 0 ... #endif. I also copied the read method signature and locals since the methods are similar. Before I get to far into the guts I should note that this effort relied on a very comprehensive spec or I would have had no idea if I was successful.

Once I implemented the initial parameter checking I dove into the String and Block (binary string) handling portion of the write method. The write method modifies the given buffer by writing a String or Block into it. I started with Google and found Chris Lalancette’s post and this excellent write-up on The Ruby C API. I also cloned ruby itself and went directly to the code. I found the code a little difficult to follow but the important thing to remember is if the method is NOT delcared static then you can use it in your C extension. I ended up using rb_str_concat to add to the buffer and rb_str_update to directly modify the buffer.

Modifying Ruby strings

If you directly modify the Ruby string buffer in a C extension with memcpy, memmove, or memset (after getting a pointer with RSTRING_PTR), you need to tell the Ruby runtime with the rb_str_modify method. Calling Ruby's methods like rb_str_update automatically handles this for you.

Another issue I ran into was the existing Ruby code was calling to_s on the input value to ensure it was a String. In the C extension you can check for a type using RB_TYPE_P(value, T_STRING) where value is a unknown Ruby VALUE instance and T_STRING can be any number of Ruby types. If the value was not a Ruby String I used rb_funcall to directly call the Ruby runtime and call the to_s method. If you are unable to find an appropriate method in the C library to do what you want, this is the way to use Ruby from within your C extension.

Next I started to tackle the writing of signed and unsigned integers. COSMOS supports overflow of integers by either truncating a passed in value, saturating to the high or low, or raising an error. So I implemented a check_overflow method in C to handle this logic. This code was very difficult to get right because of the size of the values involved. Since COSMOS handles integers of any size, I had to create Ruby Bignum constants to perform the comparisons. This involved another dive into the Ruby source to understand bignum.c. One of the tricks was to create Ruby Constants up front in the Initialization routine so I wasn’t constantly recalulating Bignums for comparison. COSMOS also handles bitfields so those values I generate dynamically using rb_big_pow and rb_bit_minus. I also created a TO_BIGNUM macro which converts Fixnum to Bignum so all the math uses Bignum methods. I could then use the rb_big_cmp to compare the given value with the appropriate minimum and maximum values.

The bitfield logic was the most complex to convert to C. This required a lot of C printfs and Ruby puts at each step of the way to ensure all the intermediary calculations were matching up. COSMOS supports big and little endian data buffers so I had to ensure the bytes were reversed and shifted as necessary before finally writing them back to the buffer. Again the rb_str_modify function is called to notify the Ruby runtime that the buffer has been changed.

The floating point values were probably the easiest because I simply called RFLOAT_VALUE(value) to get the double value of the passed in Ruby value. At this point I was able to successfully run the full spec. However, once I ran the entire COSMOS spec suite I hit a failure on a simple write call in api_spec.rb. I determined the spec was trying to send an integer value where there was a floating point value defined. The old Ruby code simply converted this value inline but I was calling RFLOAT_VALUE(value) which ASSUMES the value is a float. I updated the binary_accessor_spec.rb to capture this failure and also noted a similar issue in the integer logic. The Ruby code was calling value = Integer(value) for integers and value = Float(value) for floats. This not only handles the case of passing an integer value when you want to write a float, it also handles truncating a float to an integer and even handles parsing a String which contains a numeric value. When you have a tremendous amount of work being done by Ruby you are best to fall back to rb_funcall. But how to call the Integer() method which doesn’t appear to have a receiver. Remember that if a method doesn’t appear to have a receiver it’s probably being called on Kernel which is exactly the case. Thus I call it with value = rb_funcall(rb_mKernel, rb_intern("Float"), 1, value);. (NOTE: I also discovered I could call the method passing ‘self’ instead of rb_mKernel but using Kernel felt more explicit).

At this point I refactored to combine some of the functionality in the read method with the new write method. I probably could have done more refactoring but refactoring C code just isn’t as much fun as refactoring Ruby code. Once I completed the refactor I wanted to benchmark my new C extension to determine how much faster (or slower?) I made it. I love the benchmark-ips gem as it benchmarks iterations per second and automatically determines how many times to run the code to get good data. But I didn’t want to re-write our existing specs to support using this gem so I looked into how to integrate it with RSpec. It turns out this is all that was needed in our spec_helper.rb:

if ENV.key?("BENCHMARK") c.around(:each) do |example| Benchmark.ips do |x| x.report(example.metadata[:full_description]) do example.run end end end end

Benchmark-ips works by calculating the number of runs to get interesting data and then running the code in question. Thus defining BENCHMARK in the environment makes the specs run EXTREMELY slow. I used the ability of RSpec to filter only the examples I wanted to benchmark with the -e option:

rspec spec/packets/binary_accesor_spec.rb -e "write only"

Running this in master and then in my C-extension branch I calculated the difference in iterations and then filtered out all the “complains” (raise an exception) and “overflow” test cases to focus on just the tests which write values. The average improvement was 1.3x. Not quite as awesome as I was hoping for but an improvement in an area that is performance sensitive. I suspected I could get additional performance if I optimized the check_overflow method to not always use Bignums and to do Fixnum comparisons if possible. However, this did not yield any optimizations so I backed out the change.

At this point I submitted the pull request which broke the Travis build. Ryan then added a patch that corrected all my issues and the build passed. I re-benchmarked his changes and overall the results were actually slightly faster on average so the pull request was merged.

Enjoy a faster COSMOS write routine!

Ball Aerospace COSMOS 3.3.3 Released

Issues:

  • #93 Derived items that return arrays are not formatted to strings bug
  • #94 JsonDRb retry if first attempt hits a closed socket bug
  • #96 Make max lines written to output a variable in ScriptRunnerFrame enhancement
  • #99 Increase Block Count in DataViewer

Migration Notes from COSMOS 3.2.x:

System.telemetry.target_names and System.commands.target_names no longer contain the ‘UNKNOWN’ target.

Ball Aerospace COSMOS 3.3.1 - Startup Cheetah

COSMOS first-time startup speed is now 16 times faster - hence this release is codenamed “Startup Cheetah”. Enjoy!

Issues:

  • #91 Add mutex around creation of System.instance
  • #89 Reduce maximum block count from 10000 to 100 everywhere
  • #87 MACRO doesn’t support more than one item
  • #85 Replace use of DL with Fiddle
  • #82 Improve COSMOS startup speed
  • #81 UNKNOWN target identifies all buffers before other targets have a chance
  • #78 Reduce COSMOS memory usage
  • #76 Fix specs to new expect syntax and remove ‘should’
  • #74 Server requests/sec and utilization are incorrect

Migration Notes from COSMOS 3.2.x:

System.telemetry.target_names and System.commands.target_names no longer contain the ‘UNKNOWN’ target.

Ball Aerospace COSMOS 3.2.1 Released

Issues:

  • #61 Don’t crash TestRunner if there is an error during require_utilities()
  • #63 Creating interfaces with the same name does not cause an error
  • #64 Launcher RUBYW substitution broken by refactor
  • #65 CmdTlmServer ensure log messages start scrolled to bottom on Linux
  • #66 Improve graceful shutdown on linux and prevent continuous exceptions from InterfaceThread
  • #70 ask() should take a default

Migration Notes from COSMOS 3.1.x:

No significant updates to existing code should be needed. The primary reason for update to 3.2.x is fixing the slow shutdown present in all of 3.1.x.

Ball Aerospace COSMOS 3.2.0 Released

Issues:

  • #34 Refactor packet_config
  • #43 Add ccsds_log_reader.rb as an example of alternative log readers
  • #45 Slow shutdown of CTS and TlmViewer with threads trying to connect
  • #46 Add mutex protection to Cosmos::MessageLog
  • #47 TlmGrapher RangeError in Overview Graph
  • #49 Make about dialog scroll
  • #55 Automatic require of stream_protocol fix and cleanup
  • #57 Add OPTION keyword to support passing arbitrary options to interfaces/routers
  • #59 Add password mode to ask and ask_string

Migration Notes from COSMOS 3.1.x:

No significant updates to existing code should be needed. The primary reason for update to 3.2.x is fixing the slow shutdown present in all of 3.1.x.

Ball Aerospace COSMOS 3.1.2 Released

Issues:

  • #20 Handbook Creator should output relative paths
  • #21 Improve code metrics
  • #26 Dynamically created file for Mac launchers should not be included in CRC calculation
  • #27 TestRunner build_test_suites destroys CustomTestSuite if underlying test procedures change
  • #28 TlmGrapher - Undefined method nan? for 0:Fixnum
  • #35 Race condition starting new binary log
  • #36 TlmDetailsDialog non-functional
  • #37 Remaining TlmGrapher regression
  • #38 Allow INTERFACE_TARGET to work with target name substitutions

Migration Notes from COSMOS 3.0.x:

The definition of limits persistence has changed. Before it only applied when changing to a bad state (yellow or red). Now persistence applies for all changes including from stale to a valid state and from bad states back to green.

Ball Aerospace COSMOS 3.1.1 Released

Issues:

  • #10 Simulated Targets Button only works on Windows
  • #11 Mac application folders not working
  • #12 Persistence should be applied even if changing from stale
  • #14 Allow information on logging page to be copied
  • #16 Ensure read conversion cache cannot be cleared mid-use
  • #17 NaNs in telemetry graph causes scaling crash

Migration Notes from COSMOS 3.0.x:

The definition of limits persistence has changed. Before it only applied when changing to a bad state (yellow or red). Now persistence applies for all changes including from stale to a valid state and from bad states back to green.

Ball Aerospace COSMOS Open Sourced

Ball Aerospace & Technologies Corp. has launched Ball Aerospace COSMOS, the company’s second open source software project.

Ball Aerospace COSMOS brings an exciting set of functionality to Operations and Integration & Test that had previously only been available in proprietary and expensive COTS solutions or not available in any commercial product. A full set of 15 applications provide features including automated test procedures, realtime and offline telemetry display and graphing, post-test analysis and CSV extraction, limits monitoring, command and telemetry handbook creation, and binary file editing.

Automated test procedures written for COSMOS offer the full power of the Ruby programming language allowing operators to send commands, verify telemetry, read and write files, access the network, and even send an email on completion. Additional features include automated test report generation, standardized meta data collection (unit serial number, operator name), and loop testing (executing the same test repeatedly to wring out timing and other issues). Advanced debugging functionality allows for single-stepping through procedures, setting breakpoints, and complete logging of all script and user interaction with the system under test.

Detailed data visualization allows for custom screen creation, line and x-y plotting of realtime data, easy creation of custom 3d visualizations, and the ability to quickly view any data provided by an embedded system. Post-test analysis and data extraction capabilities make narrowing down anomalies easy and allow for data to be quickly imported into outside data analysis systems such as Matlab.

“Ball Aerospace’s COSMOS enables an amazing amount of functionality and can provide a standard interface for interacting with anything that contains embedded software”, said Ryan Melton, COSMOS’s creator and open source evangelist. “By open sourcing this software we hope to change the whole playing field of Operations, Integration, and Test”.