2014-01-16

Sample Last N Features

(FME 2014 build 14230)

Congrats FME 2014 release!

The Sampler transformer has "First N Features" option as "Sampling Type" parameter.
Well, is it possible to sample "Last N Features"?

First inspiration is: to reverse the order of input features using a Sorter, and then apply a Sampler with "First N Features" option.
It's easy, but I'm afraid that it would consume huge memory especially when there are very many features, because the Sorter will store all input features while processing. And also, if the input order of features has to be preserved, it will have to be reversed again after sampling.

Creating a custom transformer using an AttributeCreator with "Multiple Feature Attribute Support" option could be a workaround.







"SAMPLING_AMOUNT" is a Published Parameter defined in the custom transformer.
Since the back ground of "Value" setting which used the parameter with the manner as shown is colored by pink, it might be an unexpected usage. But in my test, it worked fine without any error or warning. Sometimes an unexpected usage is also effective.

Test and Result:







-----
# LastNSampler PythonCaller Edition
# Using Queue module
import fmeobjects, Queue

class LastNSampler(object):
    def __init__(self):
        # Create a FIFO (First In, First Out) queue.
        self.q = Queue.Queue(int(FME_MacroValues['SAMPLING_AMOUNT']))
     
    def pop(self, flag): # flag is a value indicating sampled or not.
        feature = self.q.get()
        feature.setAttribute('_sampled', flag)
        self.pyoutput(feature)
     
    def input(self, feature):
        if self.q.full():
            self.pop('no')
        self.q.put(feature)
     
    def close(self):
        while not self.q.empty():
            self.pop('yes')
-----
For what it's worth...

2014-01-13

Detect Invalid DateTime String with DateFormatter

(FME 2014 Beta build 14229)

"Set Invalid Date Attributes To" parameter has been added to the DateFormatter transformer in FME 2014.
In the previous version, the DateFormetter just does nothing if a datetime string is stored in invalid format. So it was necessary to use a regular expression or similar mechanism to determine whether the format is valid or not. Now it can be determined with simpler way.

This is part of a CSV table describing daily rainfall records in Tokyo, 2013.
Note: <empty> Rainfall indicates no rain, 0.0 indicates rained but the value was less than 0.05 mm/day. Those two representations have to be strictly distinguished.
DateRainfall
2013-04-010.0
2013-04-0253.5
2013-04-0360.0
2013-04-04
2013-04-050.0
2013-04-0699.5
2013-04-079.5
2013-04-08
2013-04-09
2013-04-10

The correct date format is "%Y-%m-%d". Assume that I need to validate and change date format for the following processing - e.g. writing the records into a database table.

I think a combination of a DateFormatter and a Tester can be used effectively in this case. That is to say, a Tester can be used to detect invalid records easily if the "Set Invalid Date Attributes To" parameter is specified like this.
=====
2014-01-19: Of course an invalid date string can be replaced with <null> in the DateFormatter, and also the Tester can be used to determine whether an attribute value is <null>.  Using <null> might be more intuitive than replacing with specific string value.
=====

Adding a NullAttributeMapper
When the MDB Writer receives an <empty> Rainfall, it would write <null> into the associated field in the destination table, because non-numeric attribute value (including empty string) cannot be converted to a decimal type value. Then the writer logs a warning message like this.

MS Access Writer: 179 attribute value(s) failed conversion, and were written as NULL values

Although the destination table would be written as expected, I would insert a NullAttributeMapper between the Tester and the Writer in order to convert <empty> to <null> explicitly, so that other more important warnings which have to be detected will not be overlooked.
I think it would be one of general NullAttributeMapper use cases.

2014-01-11

xfMap: Generate Common Feature Type for Different Schemas

Many fundamental geographic datasets are provided by Japanese government in XML format.
> 基盤地図情報 Fundamental Geospatial Data
> 国土数値情報 National  Land Numerical Information (English)
Naturally, XML schema for every data type is strictly defined, but in fact, there are some datasets created in different (incorrect) schema.

This is a simplified example describing administrative areas in certain year. The actual data contains some more attributes and geometry elements (and of course contains Japanese characters), but those are omitted for simplifying. Assume this is the correct schema.
-----
<?xml version="1.0"?>
<Dataset>
    <AdministrativeBoundary>
        <prefectureName>Chiba</prefectureName>
        <cityName>Matsudo</cityName>
        <administrativeAreaCode>12207</administrativeAreaCode>
    </AdministrativeBoundary>
    <AdministrativeBoundary>
        <prefectureName>Chiba</prefectureName>
        <cityName>Kashiwa</cityName>
        <administrativeAreaCode>12217</administrativeAreaCode>
    </AdministrativeBoundary>
</Dataset>
-----
# I'm living in Kashiwa city :-)

This is the same area data created in a different year. Contents are same as above, but schema is incorrect.
-----
<?xml version="1.0"?>
<Dataset>
    <AdministrativeArea>
        <prn>Chiba</prn>
        <cn2>Matsudo</cn2>
        <acc>12207</acc>
    </AdministrativeArea>
    <AdministrativeArea>
        <prn>Chiba</prn>
        <cn2>Kashiwa</cn2>
        <acc>12217</acc>
    </AdministrativeArea>
</Dataset>
-----

I need to read both datasets as same feature type with an XML Reader. Although those schemas are simple, the XML Reader with "Feature Paths" option cannot generate a common feature type from different schemas. How should I do that?

I solved the issue by defining an xfMap like this.
-----
<?xml version="1.0"?>
<xfMap>
    <feature-map>
        <!-- For correct schema -->
        <mapping match="AdministrativeBoundary">
            <feature-type>
                <literal expr="AdministrativeBoundary" />
            </feature-type>
            <structure matched-prefix="no" cardinality="*/+" />
        </mapping>

        <!-- For incorrect schema -->
        <mapping match="AdministrativeArea">
            <feature-type>
                <literal expr="AdministrativeBoundary" />
            </feature-type>
            <attributes>
                <attribute>
                    <name><literal expr="prefectureName" /></name>
                    <value><extract expr="./prn" /></value>
                </attribute>
                <attribute>
                    <name><literal expr="cityName" /></name>
                    <value><extract expr="./cn2" /></value>
                </attribute>
                <attribute>
                    <name><literal expr="administrativeAreaCode" /></name>
                    <value><extract expr="./acc" /></value>
                </attribute>
            </attributes>
        </mapping>
    </feature-map>
</xfMap>
-----

The first <mapping> element is for the correct schema. The functionality is similar to flattening <AdministrativeBoundary> element with "Feature Paths" option.
The second <mapping> element will map XML element values described in incorrect elements to the correct schema.

The XML Reader with the xfMap generates a feature type named "AdministrativeBoundary" like this (FME 2014 Beta build 14227). It can read any dataset created in both correct schema and incorrect schema.











I'm not sure why the government published the data with incorrect schema, and I'm wondering that nobody has noticed that. In fact, the government is also providing datasets in Esri Shape format simultaneously, so I suspect nobody uses XML data. But some parts of the original XML data cannot be stored in Shape datasets by limitation of the format specification. I think it is a loss to the nation if those cannot be used effectively.
FME can read them flexibly with no loss even though the schema is not correct. That's great.

In addition, <schema-type> element can be added to the xfMap to define data types of attributes and the order of them explicitly. For example:
-----
<?xml version="1.0"?>
<xfMap>
    <schema-type>
        <inline>
            <schema-feature type="AdministrativeBoundary">
                <schema-attribute name="fme_geometry{0}" type="xml_no_geom" />
                <schema-attribute name="prefectureName" type="xml_buffer" />
                <schema-attribute name="cityName" type="xml_buffer" />
                <schema-attribute name="administrativeAreaCode" type="xml_buffer" />
            </schema-feature>
        </inline>
    </schema-type>
    <feature-map>
        ...
    </feature-map>
</xfMap>
-----

2014-01-10

Create Raster from Serialized Cell Values

(FME 2014 Beta build 14227)

I have a lot of DEM datasets, each of them contains grid cell values (altitudes) in a standard mesh, i.e. rectangular area.
Mesh size, number of columns and number of rows per a mesh are fixed as followings.
- mesh width = 45 seconds
- mesh height = 30 seconds
- number of rows = 150
- number of columns = 225
Therefore the number of cells is 33750 (= 150 x 225) per a mesh and cell size is 0.2x0.2 seconds.
Cell values will be stored in a list attribute named "_alti{}" - an array containing sequenced altitude values from south-west cell (0, 0) to north-east cell (149, 224). And also each mesh feature has the coordinate of its south-west corner as attributes named "_xmin" and "_ymin".

Consider creating a raster geometry for each mesh in such condition.
My strategy is:
1) Initially create a raster whose origin is (0, 0) and cell size is (1 x 1).
2) Scale the raster by appropriate ratio.
3) Move the raster so that the origin is located at the correct coordinate.

At first, extract cell values with a ListExploder, then create cell center points in 3D with a VertexCreator (2D/3DPointReplacer/Adder have been integrated into it). Since the cell size of initial raster is (1 x 1), the expressions for calculating (x, y) of each point are simple.
And then, create a raster with a NumericRasterizer. The parameter settings are also simple.
After creating raster, recover the original attributes using a FeatureMerger.















In the next Bookmark, scale the raster by appropriate ratio using a Scaler and move it to correct location using an Offsetter. Done!










Of course the raster can be created by the final cell size and location from the beginning. But in my trial, the way above was a little more efficient in running speed, even though having additional transformers for scaling and offsetting.
Above all, I think it's nice that calculation and parameter settings are very simple.

P.S. I feel the "VertexCreator and NumericRasterizer" method is not so efficient, but it seems to be the only way to create a raster geometry based on serialized grid cell values using existing transformers only. I looked for a way to create a raster more efficiently by Python scripting, but couldn't find functions to do that in FME Objects Python API unfortunately.
Note (2014-01-14): Inserting a PointCloudCombiner transformer between the VertexCreator and the NumericRasterizer could be effective to improve efficiency as mentioned below.

=====
2014-01-11: There are examples in which "Scaler and Offsetter" method can be used effectively.
> Community: Shrink / fit an object to another
> Community: Scale by a point
> Advanced Geometric Operation: Create Normal Lines of a Line Segment
=====
2014-01-14: Considerations on improving efficiency
1) Create 3D Points using Python script
As mentioned above FME Objects Python API doesn't provide functions to create a raster. But 3D points can be created easily with Python, and also  it could be more efficient. For example, the ListExploder and the VertexCreator can be replaced with a PythonCaller performing this script.
-----
# Example 1: Create 3D Points
import fmeobjects
class GridCellPointCreator(object):
    def __init__(self):
        pass
     
    def input(self, feature):
        id = feature.getAttribute('_feature_id')
        for i, a in enumerate(feature.getAttribute('_alti{}')):
            x, y, z = i % 225 + 0.5, i / 225 + 0.5, float(a)
            point = fmeobjects.FMEFeature()
            point.setGeometry(fmeobjects.FMEPoint(x, y, z))
            point.setAttribute('_feature_id', id)
            self.pyoutput(point)
     
    def close(self):
        pass
-----
Depending on the condition, creating a 3D multi point geometry could be more efficient.
-----
# Example 2: Create 3D Multi Point
# Unnecessary attributes should be removed before rasterizing.
import fmeobjects
def createGridCellPoints(feature):
    points = fmeobjects.FMEMultiPoint()
    for i, a in enumerate(feature.getAttribute('_alti{}')):
        x, y, z = i % 225 + 0.5, i / 225 + 0.5, float(a)
        points.appendPart(fmeobjects.FMEPoint(x, y, z))
    feature.setGeometry(points)
-----
2) Transform Vector Geometry to a Point Cloud before rasterizing
The NumericRasterizer can also create a raster based on a point cloud. In my trial, transforming 3D points (or 3D multi point) to a point cloud using a PointCloudCombiner transformer before creating a raster was indeed more efficient. Furthermore, since the PointCloudCombiner can transform any geometry other than points to a point cloud, creating a 3D line geometry with Python could be also an option.
Dale and Chris suggested to me the capability of the point cloud technology. Thanks!
=====
2014-01-15
-----
# Example 3: Create 3D Line
import fmeobjects
def create3DLine(feature):
    coords = []
    for i, a in enumerate(feature.getAttribute('_alti{}')):
        coords.append((i % 225 + 0.5, i / 225 + 0.5, float(a)))
    feature.setGeometry(fmeobjects.FMELine(coords))
-----

2014-01-04

Null in FME 2014: Magnitude Relationship

(FME 2014 Beta build 14223)

Happy New Year!
I'll continue to explore the <null> world. My first question in this year is:
Are there differences in magnitude among "nothing" values - <null>, <empty> and <missing>?

The following image is the result of testing with the Sorter. Each input feature has an attribute named "value" which holds <null> or <empty>, or is missing the attribute (i.e. <missing>), and "index" attribute indicates the input order.
Left table shows the input order; right table shows the result of sorting by "value". As the image showing, the Sorter had no influences on the order of features; it seems that there is no difference in magnitude among them, at least in sorting.













Then, I tried cases that there are features having "non-nothing" value as the attribute.

The result of Alphabetic ascending sorting is like this image.
All the "nothing" values is less than any "non-nothing" value. It's as expected.













The next image shows the result of Numeric ascending sorting.
All the "nothing" values seems to be treated as equivalent to 0 in numeric sorting.













The results are similar to the result of sorting non-empty values and empty strings in FME 2013.
I think the Sorter is keeping logical consistency with the previous version and other transformers - e.g. the Tester etc.. But the behavior especially in numeric sorting may look strange intuitively. There would be some cases in which we should mind that.

=====
2014-01-06: The help documentation on the Sorter in FME 2014 (build 14223) describes like this.
"In Alphabetic mode, missing and null attribute values are treated as the empty string when they are sorted. In Numeric mode, non-numeric attribute values are treated as the value 0 when they are sorted. For example, missing, null, the empty string, and the string "apple" are all treated as the value 0 in Numeric mode."
=====

Tip: If it's necessary to distinguish each "nothing" type in sorting, create another attribute which represents the type of value, so that features can be sorted by the type and value.