2014-10-12

Cases where Null and Missing are considered as Empty String

Null(ナル値)、Missing(属性の欠落)が Empty String(空文字列)と見なされるケース

As you know, Null has been introduced in FME 2014. It has brought some big benefits to us.
In FME 2014+, many transformers can distinguish Null, Missing, and Empty String strictly.
On the other hand, there also are several cases where Null and Missing are considered as Empty String.
ご存じの通りFME 2014ではNull(ナル値)が導入され、大いに役立っています。
FME 2014以降では、多くのトランスフォーマーはNull, Missing(属性の欠落)及びEmpty String(空文字列) を厳密に区別することができますが、一方では、Null, Missingが空文字列とみなされるケースもあります。

A typical case is the = operator in a test condition setting for a transformer.
For example, the Tester with this setting sends a feature which stores Null as "attr" or doesn't have the "attr", to the Passed port.
It doesn't distinguish both Null and Missing from Empty String.
テスト条件における = 演算子は典型的なケースです。
例えば、次のような設定をしたTesterは、属性"attr"にNullを保持する、あるいは、"attr"を持たないフィーチャーをPassedポートから出力します。つまり、NullやMissingを空文字列と区別しません。
=====
2014-10-14: Some users may feel this behavior strange, but I think it's reasonable as a general manner in FME processing. Dale@Safe added some explanations about the basic concept and the background. See also his comment.
この振る舞いを奇妙に思う方もいらっしゃるかも知れませんが、私は、FMEの処理における一般的な方法として合理的であると思います。Safe社のデールさんが基本的な考え方と背景について説明を加えてくださいました。彼のコメントも参照してください。
=====














If you need to test whether the attribute value is Null or the attribute is Missing, you should use "Attribute Is Null" and "Attribute Is Missing" operators, like this.
属性値がNullであるかどうか、あるいは、属性が欠落しているかどうかを調べる必要がある場合は、次のように Attribute Is Null 演算子や Attribute Is Missing 演算子を使う必要があります。














As well, the Group By parameter does not distinguish them.
If you set an attribute to Group By parameter of a transformer, the transformer will configure one group for every feature which holds Null or Empty String as the attribute, or doesn't have the attribute (i.e. Missing).
There may be some users who feel such a behavior strange. However, it seems to be related to the fundamental mechanism regarding parameters, so I don't think it will change in the future.
* See also the following Note (2014-12-09).
同様に、Group Byパラメーターにおいてもそれらは区別されません。
Group Byパラメーターにある属性が設定されたトランスフォーマーは、その属性にナル値または空文字列を格納する、あるいは、その属性を持たない全てのフィーチャーをひとつのグループとして取り扱います。この振る舞いを奇妙に感じるユーザーもいるかも知れませんが、これはパラメーターの基本的なメカニズムに関係しているようであり、今後もこれが変わることはないと思います。
* 次の注(2014-12-09)もご覧ください。
=====
Note (2014-12-09): Got a great information. I heard that Safe is going to improve the behavior of Group By concerning Null, Missing, and Empty. Now, I update the last paragraph above to:
"Although it seems to be related to the fundamental mechanism regarding parameters, it might be improved in the future."
注 (2014-12-09): 素晴らしい情報を得ました。Safe社は、Null, Missing, 空文字列に関するGroup Byの振る舞いについて改良する予定であるとのことです。上の最後の段落を次のように更新します。
これはパラメーターの基本的なメカニズムに関係しているようですが、将来は改良される可能性があります。
=====

Well, how can you make groups separately for Null, Missing, and Empty String?
A possible way I can think of is to assign special values to identify "Null" and "Missing" with the NullAttributeMapper before grouping.
Of course the special values must not be equal to any other possible values for the attribute, but it's not so difficult in many cases, I think.
では、Null、Missing、Empty Stringを区別してグループ化するにはどうすれば良いでしょう?
考えられる方法は、グループ化する前に NullAttributeMapper を使ってNull, Missingを識別するための特別な値を割り当てることです。もちろんその特別な値は、その属性の値として可能な全ての他の値と異ならなければなりませんが、多くの場合、それはそんなに難しいことではないと思います。

Anyway, we need to know that there are cases where Null and Missing are considered as Empty String.
ともあれ、Null, Missing が空文字列として扱われるケースがあることは知っておく必要があります。

2 comments:

  1. Hi Takashi,

    The reason for the behaviour in FME with respect to missing and NULL being considered the same as empty goes back to the original design more than 20 years ago. For the situations we typically saw, there was not sufficient value in distinguishing those cases for the simplicity that came from treating them the same. So that is why, in general, unless NULL and Missing are specifically tested for, we will treat them as if they are a blank value.

    We feel that this is a reasonable compromise that can keep the past 20 years of workspaces/mapping files working the same as they always did, and yet allow for those that need to make this distinction between NULL, Missing, Empty, able to do so. But the general rule to keep in mind is that, unless specifically called out, NULL and Missing will be considered the same as Empty.

    I apologize for this but as with many things in FME, the situation we have we feel is the best balance between the Perfect and the Good.

    Dale

    ReplyDelete
    Replies
    1. Hi Dale,

      Thank you for the comment.
      I agree that it's reasonable that Null and Missing are generally treated as an Empty String. I'm just afraid that FME beginners may get confused, since the concept is not so clear in FME documentations. e.g. the description on = operator in the help of Tester doesn't mention about this, in FME 2014.

      In FME 2015, however, the help has been enhanced wonderfully:
      "When the comparison operators =, !=, <, >, <=, >= are applied to null, missing, and empty string values, the trio will all evaluate to the empty string. Thus, they are pairwise equal in the context of these comparison operators. Further, the following comparison ordering is respected:
      Null, missing, empty string < numbers < other values"

      I really appreciate the enhancement.

      Takashi

      Delete