\documentclass[twoside,11pt]{article}

\usepackage{jmlr2e}
\usepackage{times}
\usepackage{mathptm}
\usepackage{amssymb}

\jmlrheading{5}{2004}{153-188}{12/02}{2/04}{Nada Lavra\v{c}, Branko
Kav\v{s}ek, Peter Flach, and Ljup\v{c}o Todorovski}


\ShortHeadings{Subgroup discovery with CN2-SD}{Lavra\v{c} et al.}
\firstpageno{153}

\usepackage{epsf}


%-------------------------------------------------------------------------
\input{psfig.sty}

\begin{document}

\title{Subgroup Discovery with CN2-SD}

\author{\name Nada Lavra\v{c} \email nada.lavrac@ijs.si \\
       \addr Jo\v{z}ef Stefan Institute\\ 
Jamova 39\\ 
1000 Ljubljana, Slovenia and \\
           \addr Nova Gorica Polytechnic\\
 Vipavska 13\\
 5100 Nova\\
           Gorica, Slovenia
       \AND
       \name Branko Kav\v{s}ek \email branko.kavsek@ijs.si \\
       \addr Jo\v{z}ef Stefan Institute\\
 Jamova 39\\
 1000 Ljubljana, Slovenia
       \AND
       \name Peter Flach \email peter.flach@bristol.ac.uk \\
       \addr University of Bristol\\
 Woodland Road\\
 Bristol BS8 1UB, United Kingdom
       \AND
       \name Ljup\v{c}o Todorovski \email ljupco.todorovski@ijs.si \\
       \addr Jo\v{z}ef Stefan Institute\\
 Jamova 39\\
 1000 Ljubljana, Slovenia}

\editor{Stefan Wrobel}

\maketitle

\begin{abstract}
  This paper investigates how to adapt standard classification rule
  learning approaches to subgroup discovery. The goal of subgroup
  discovery is to find rules describing subsets of the population
  that are sufficiently large and statistically unusual. The paper
  presents a subgroup discovery algorithm, {\em CN2-SD}, developed
  by modifying parts of the CN2 classification rule learner: its
  covering algorithm, search heuristic, probabilistic classification
  of instances, and evaluation measures.  Experimental evaluation of
  {\em CN2-SD} on 23 UCI data sets shows substantial reduction of
  the number of induced rules, increased rule coverage and rule
  significance, as well as slight improvements in terms of the area
  under ROC curve, when compared with the CN2 algorithm. Application
  of {\em CN2-SD} to a large traffic accident data set confirms
  these findings.
\end{abstract}


\begin{keywords}
  Rule Learning, Subgroup Discovery, UCI Data Sets, Traffic Accident
  Data Analysis
\end{keywords}

\section{Introduction}
\label{sec-intro}

Rule learning is most frequently used in the context of classification
rule learning \citep{Michalskietal.1986,cla:nib:1989,cohen1995} and
association rule learning \citep{Agrawaletal.1996}. While
classification rule learning is an approach to {\em predictive
induction} (or supervised learning), aimed at constructing a set of
rules to be used for classification and/or prediction, association
rule learning is a form of {\em descriptive induction}
(non-classificatory induction or unsupervised learning), aimed at the
discovery of individual rules which define interesting patterns in
data.

Descriptive induction has recently gained much attention of the rule
learning research community.  Besides mining of association rules
(e.g., the APRIORI association rule learning algorithm
\citep{Agrawaletal.1996}), other approaches have been developed,
including clausal discovery as in the CLAUDIEN system
\citep{DeRaedt1997,DeRaedt2001}, and database dependency discovery
\citep{FlachandSavnik1999}.

\subsection{Subgroup Discovery: A Task at the Intersection of
  Predictive and Descriptive Induction}

This paper shows how classification rule learning can be adapted to
{\em subgroup discovery}, a task at the intersection of predictive and
descriptive induction, that has first been formulated by 
\citet{Klosgen1996} and \citet{Wrobel1997,Wrobel2001}, and addressed by rule
learning algorithms EXPLORA \citep{Klosgen1996} and MIDOS
\citep{Wrobel1997,Wrobel2001}.  In the work of \citet{Klosgen1996} and
\citet{Wrobel1997,Wrobel2001}, the problem of subgroup
discovery has been defined as follows: Given a population of
individuals and a property of those individuals we are interested in,
find population subgroups that are statistically `most interesting',
e.g., are as large as possible and have the most unusual statistical
(distributional) characteristics with respect to the property of
interest.

In subgroup discovery, rules have the form $Class \leftarrow
Cond$, where the property of interest for subgroup discovery is
class value $Class$ that appears in the rule consequent, and the
rule antecedent $Cond$ is a conjunction of features
(attribute-value pairs) selected from the features describing the
training instances. As rules are induced from labeled training
instances (labeled positive if the property of interest holds, and
negative otherwise), the process of subgroup discovery is targeted
at uncovering properties of a selected {\em target} population of
individuals with the given property of interest. In this sense,
subgroup discovery is a form of supervised learning. However, in
many respects subgroup discovery is a form of descriptive
induction as the task is to uncover individual interesting {\em
patterns} in data. The standard assumptions made by classification
rule learning algorithms (especially the ones that take the
covering approach), such as `induced rules should be as accurate
as possible' or `induced rules should be as distinct as possible,
covering different parts of the population', need to be relaxed.
In our approach, the first assumption, implemented in
classification rule learners by heuristic which aim at optimizing
predictive accuracy, is relaxed by implementing new heuristics for
subgroup discovery which aim at finding `best' subgroups in terms
of rule coverage and distributional unusualness. The relaxation of
the second assumption enables the discovery of overlapping
subgroups, describing some population segments in a multiplicity
of ways. Induced subgroup descriptions may be redundant, if viewed
from a classifier perspective, but very valuable in terms of their
descriptive power, uncovering genuine properties of subpopulations
from different viewpoints.

Let us emphasize the difference between subgroup discovery (as a
task at the intersection of predictive and descriptive induction)
and classification rule learning (as a form of predictive
induction). The goal of standard rule learning is to generate
models, one for each class, consisting of rule sets describing
class characteristics in terms of properties occurring in the
descriptions of training examples. In contrast, subgroup discovery
aims at discovering individual rules or `patterns' of interest,
which must be represented in explicit symbolic form and which must
be relatively simple in order to be recognized as actionable by
potential users. Moreover, standard classification rule learning
algorithms cannot appropriately address the task of subgroup
discovery as they use the covering algorithm for rule set
construction which - as will be seen in this paper - hinders the
applicability of classification rule induction approaches in
subgroup discovery.

Subgroup discovery is usually seen as different from classification,
as it addresses different goals (discovery of interesting population
subgroups instead of maximizing classification accuracy of the
induced rule set). This is manifested also by the fact that
in subgroup discovery one can often tolerate many more false
positives (negative examples incorrectly classified as positives) than
in a classification task. However, both tasks, subgroup discovery and
classification rule learning, can be unified under the umbrella of
cost-sensitive classification. This is because when deciding which
classifiers are optimal in a given context it does not
matter whether we penalize false negatives as is the case in
classification, or reward true positives as in subgroup discovery.

\subsection{Overview of the {\em CN2-SD} Approach to Subgroup Discovery}

This paper investigates how to adapt standard classification rule
learning approaches to subgroup discovery. The proposed modifications
of classification rule learners can, in principle, be used to modify
any rule learner using the covering algorithm for rule set construction. In
this paper, we illustrate the approach by modifying the well-known
CN2 rule learning algorithm \citep{cla:nib:1989,cla:bos:1991}.
Alternatively, we could have
modified RL \citep{Leeetal.1998}, RIPPER \citep{cohen1995}, SLIPPER
\citep{cohen1999} or other more sophisticated classification rule
learners. The reason for modifying CN2 is that other more
sophisticated learners include advanced techniques that make them more
effective in classification tasks, improving their classification
accuracy. Improved classification accuracy is, however, not of
ultimate interest for subgroup discovery, whose main goal is to find
interesting population subgroups.

We have implemented the new subgroup discovery algorithm {\em CN2-SD}
by modifying CN2 \citep{cla:nib:1989,cla:bos:1991}.  The proposed
approach performs subgroup discovery through the following
modifications of CN2: (a) replacing the accuracy-based search
heuristic with a new weighted relative accuracy heuristic that trades
off generality and accuracy of the rule, (b) incorporating example
weights into the covering algorithm, (c) incorporating example weights
into the weighted relative accuracy search heuristic, and (d) using
probabilistic classification based on the class distribution of
covered examples by individual rules, both in the case of unordered
rule sets and ordered decision lists. In addition, we have extended the
ROC analysis framework to subgroup discovery and propose a set of
measures appropriate for evaluating the quality of induced subgroups.

This paper presents the {\em CN2-SD} subgroup discovery algorithm,
together with its experimental evaluation on 23 data sets of the UCI
Repository of Machine Learning Databases \citep{mur:aha:1994}, as
well as its application to a real world problem of traffic
accident analysis. The experimental comparison with CN2 demonstrates
that the subgroup discovery algorithm {\em CN2-SD} produces
substantially smaller rule sets, where individual rules have
higher coverage and significance. These three factors are
important for subgroup discovery: smaller size enables better
understanding, higher coverage means larger support, and higher
significance means that rules describe discovered subgroups that
have significantly different distributional characteristics
compared to the entire population. The appropriateness for
subgroup discovery is confirmed also by slight improvements in
terms of the area under ROC curve, without decreasing
predictive accuracy.

The paper is organized as follows. Section~\ref{sec-CN2} introduces
the background of this work which includes the description of the CN2
rule learning algorithm, the weighted relative accuracy heuristic, and
probabilistic classification of new examples.
Section~\ref{sec-CN2-SD} presents the subgroup discovery algorithm
{\em CN2-SD} by describing the necessary modifications of CN2.  In
Section~\ref{sec-ROC} we discuss subgroup discovery from the
perspective of ROC analysis. Section~\ref{sec-evaluation} presents a
range of metrics used in the experimental evaluation of {\em CN2-SD}.
Section~\ref{sec-experiments} presents the results of experiments on
selected UCI data sets as well as an application of {\em CN2-SD} on
a real-life traffic accident data set. Related work is discussed in
Section~\ref{sec-related}. Section~\ref{sec-conclusions} concludes
by summarizing the main contributions and proposing directions for
further work.


\section{Background}
\label{sec-CN2}

This section presents the background of our work: the classical
CN2 rule induction algorithm, including the covering algorithm for
rule set construction, the standard CN2 heuristic, weighted relative
accuracy heuristic, and the probabilistic classification technique
used in CN2.

\subsection{The CN2 Rule Induction Algorithm}
\label{sec-2.1}

CN2 is an algorithm for inducing propositional classification rules
\citep{cla:nib:1989,cla:bos:1991}. Induced rules have the form ``{\tt if}
$Cond$ {\tt then} $Class$'', where $Cond$ is a conjunction of features
(pairs of attributes and their values) and $Class$ is the class
value. In this paper we use the notation $Class \leftarrow Cond$.

CN2 consists of two main procedures: the bottom-level search procedure
that performs beam search in order to find a single rule, and the
top-level control procedure that repeatedly executes the bottom-level
search to induce a rule set. The bottom-level performs beam
search\footnote{CN2 constructs rules in a general-to-specific fashion,
  specializing only the rules in the beam (the best rules) by
  iteratively adding features to condition $Cond$. This procedure
  stops when no specialized rule can be added to the beam, because
  none of the specializations is more accurate than the rules in the
  beam.}
using classification accuracy of the rule as a heuristic function. The
accuracy of a propositional classification rule of the form $Class
\leftarrow Cond$ is equal to the conditional probability of class
$Class$, given that condition $Cond$ is satisfied:

$$Acc(Class \leftarrow Cond) = p(Class|Cond) =
\frac{p(Class.Cond)}{p(Cond)}.$$

  \noindent Usually, this probability is estimated by relative
  frequency $\frac{n(Class.Cond)}{n(Cond)}$.\footnote{Here we use the
    following notation: $n(Cond)$ stands for the number of instances
    covered by rule $Class \leftarrow Cond$, $n(Class)$ stands for the
    number of examples of class $Class$, and $n(Class.Cond)$ stands
    for the number of correctly classified examples (true positives).
    We use $p(\ldots)$ for the corresponding probabilities.} Different
  probability estimates, like the Laplace \citep{cla:bos:1991} or the
  $m$-estimate \citep{ces:1990,Dzeroskietal.1993}, can be used in CN2
  for estimating the above probability. The standard CN2 algorithm
  used in this work uses the Laplace estimate, which is computed as
  $\frac{n(Class.Cond)+1}{n(Cond)+k}$, where $k$ is the number of
  classes (for a two-class problem, $k=2$).

  CN2 can also apply a significance test to an induced rule. A rule is
  considered to be significant, if it expresses a regularity unlikely
  to have occurred by chance. To test significance, CN2 uses the
  likelihood ratio statistic \citep{cla:nib:1989} that measures the
  difference between the class probability distribution in the set of
  training examples covered by the rule and the class probability
  distribution in the set of all training examples (see
  Equation~\ref{eq:sig} in Section~\ref{sec-evaluation}). The
  empirical evaluation in the work of \citet{cla:bos:1991} shows that applying the
  significance test reduces the number of induced rules at a cost of
  slightly decreased predictive accuracy.

  Two different top-level control procedures can be used in CN2. The
  first induces an ordered list of rules and the second an unordered
  set of rules. Both procedures add a default rule (providing for
  majority class assignment) as the final rule in the induced rule set.
  When inducing an ordered list of rules, the search procedure looks
  for the most accurate rule in the current set of training examples.
  The rule predicts the most frequent class in the set of covered
  examples. In order to prevent CN2 finding the same rule again, all
  the covered examples are removed before a new iteration is started
  at the top-level. The control procedure invokes a new search, until
  all the examples are covered or no significant rule can be found. In
  the unordered case, the control procedure is iterated, inducing
  rules for each class in turn. For each induced rule, only covered
  examples belonging to that class are removed, instead of removing
  all covered examples, like in the ordered case. The negative
  training examples (i.e., examples that belong to other classes)
  remain.

\subsection{The Weighted Relative Accuracy Heuristic}
\label{sec-2.2}

Weighted relative accuracy \citep{lav:fla:zup:1999,tod:fla:lav:2000}
is a variant of rule accuracy that can be applied both in the
descriptive and predictive induction framework; in this paper
this heuristic is applied for subgroup discovery. Weighted relative
accuracy, a reformulation of one of the heuristics used in EXPLORA
\citep{Klosgen1996} and MIDOS \citep{Wrobel1997}, is defined as
follows:

\vspace*{-0.2cm}
\begin{equation}
\label{eq:wracc}
{\it \textit{WRAcc}}(Class \leftarrow Cond) = p(Cond) \cdot(p(Class|Cond) -
p(Class)).
\end{equation}

\noindent
Like most other heuristics used in subgroup discovery systems,
weighted relative accuracy consists of two components, providing a
tradeoff between rule {\it generality} (or the relative size of a
subgroup $p(Cond)$) and distributional unusualness or {\it relative
accuracy} (the difference between rule accuracy $p(Class|Cond)$ and
default accuracy $p(Class)$). This difference can also be seen as the
accuracy gain relative to the fixed rule $Class \leftarrow true$,
which predicts that all instances belong to $Class$: a rule is
interesting only if it improves upon this `default' accuracy. Another
aspect of relative accuracy is that it measures the difference
between true positives and the expected true positives (expected
under the assumption of independence of the left and right hand-side
of a rule), i.e., the utility of connecting rule body $Cond$ with
a given rule head $Class$. However, it is easy to
obtain high relative accuracy with highly specific rules, i.e., rules
with low generality $p(Cond)$. To this end, generality is used as a
`weight', so that weighted relative accuracy trades off generality of
the rule ($p(Cond)$, i.e., rule coverage) and relative accuracy
($p(Class|Cond) - p(Class)$).

In the work of \citet{Klosgen1996}, these quantities are referred to as $g$
(generality), $p$ (rule accuracy or precision) and $p_0$ (default
rule accuracy) and different tradeoffs between rule
generality and precision in the so-called $p$-$g$
(precision-generality) space are proposed. In addition to
function $g(p-p_0)$, which is equivalent to our weighted relative
accuracy heuristic, other tradeoffs that reduce the influence of
generality are proposed, e.g., $\sqrt{g}(p-p_0)$ or
$\sqrt{{g / (1-g)}}(p-p_0)$. Here, we favor the weighted
relative accuracy heuristic, because it has an intuitive
interpretation in ROC space, discussed in
Section~\ref{sec-ROC}.

\subsection{Probabilistic Classification}

The induced rules can be ordered or unordered. Ordered rules are
interpreted as a decision list \citep{riv:1987} in a straightforward
manner: when classifying a new example, the rules are sequentially
tried and the first rule that covers the example is used for
prediction.

In the case of unordered rule sets, the distribution of covered
training examples among classes is attached to each rule. Rules of the
form:

\vspace*{-0.2cm}
\begin{center}
{\tt if} $Cond$ {\tt then} $Class$ [$ClassDistribution$]
\end{center}
\vspace*{-0.2cm}

\noindent are induced, where numbers in the $ClassDistribution$ list
denote, for each individual class, how many training examples of this
class are covered by the rule.  When classifying a new example, all
rules are tried and those covering the example are collected. If a
clash occurs (several rules with different class predictions cover the
example), a voting mechanism is used to obtain the final prediction:
the class distributions attached to the rules are summed to determine
the most probable class.  If no rule fires, the default rule is
invoked to predict the majority class of training instances not covered
by the other rules in the list.

Probabilistic classification is illustrated on a sample classification
task, taken from \citet{cla:bos:1991}. Suppose we need to classify an
animal which is a two-legged, feathered, large, non-flying and has a
beak,\footnote{The animal being classified is a weka.}, and the
classification is based on a rule set, listed in
Table~\ref{tab-ruleset-birds} formed of three probabilistic rules
with the [bird, elephant] class distribution assigned to each rule
(for simplicity, the rule set does not include the default rule).
All rules fire for the animal to be classified, resulting
in a [35,10] class distribution. As a result, the animal is classified
as a bird.

\begin{table}[t]
\begin{center}
\vspace*{0.5cm}
\begin{tabular}{lllllll}
{\tt if} & legs = 2   & \& & feathers = yes & {\tt then}  & class = bird     & [13,0] \\
{\tt if} & beak = yes &    &                & {\tt then}  & class = bird     & [20,0] \\
{\tt if} & size = large & \& & flies = no   &  {\tt then} & class = elephant & [2,10]
\end{tabular}
\caption{A rule set consisting of two rules for class `bird' and one rule for class `elephant'.
\label{tab-ruleset-birds}}
\end{center}
\end{table}

\section{The {\em CN2-SD} Subgroup Discovery Algorithm}
\label{sec-CN2-SD}

The main modifications of the CN2 algorithm, making it appropriate for
subgroup discovery, involve the implementation of the weighted
covering algorithm, incorporation of example weights into the weighted
relative accuracy heuristic, probabilistic classification also in the
case of the `ordered' induction algorithm, and the area under ROC
curve rule set evaluation. This section describes the CN2
modifications, while ROC analysis and  a novel
interpretation of the weighted relative accuracy heuristic in ROC
space are given in Section~\ref{sec-ROC}.

\subsection{Weighted Covering Algorithm}

If used for subgroup discovery, one of the problems of standard rule
learners, such as CN2 and RIPPER, is the use of the covering algorithm
for rule set construction. The main deficiency of the covering algorithm
is that only the first few induced rules may be of interest as subgroup
descriptions with sufficient coverage and significance. In the
subsequent iterations of the covering algorithm, rules are induced
from biased example subsets, i.e., subsets including only positive
examples that are not covered by previously induced rules, which
inappropriately biases the subgroup discovery process.

As a remedy to this problem we propose the use of a weighted covering
algorithm \citep{GambergerLavrac2002}, in which the subsequently
induced rules (i.e., rules induced in the later stages) also
represent interesting and sufficiently large subgroups of the
population. The weighted covering algorithm modifies the classical
covering algorithm in such a way that covered positive examples are
not deleted from the current training set. Instead, in each run of the
covering loop, the algorithm stores with each example a count
indicating how often (with how many rules) the example has been
covered so far. Weights derived from these example counts then appear
in the computation of {\em \textit{WRAcc}}. Initial weights of all positive
examples $e_j$ equal 1, $w(e_j,0) = 1$. The initial example weight
$w(e_j,0) = 1$ means that the example has not been covered by any
rule, meaning `please cover this example, since it has not been
covered before', while lower weights, $ 0 < w < 1 $ mean `do not try
too hard on this example'. Consequently, the examples already covered
by one or more constructed rules decrease their weights while the
uncovered target class examples whose weights have not been
decreased will have a greater chance to be covered in the following
iterations of the algorithm.

For a weighted covering algorithm to be used, we have to
specify the weighting scheme, i.e., how the weight of each
example decreases with the increasing number of covering rules.
We have implemented two weighting schemes described below.

\subsubsection{Multiplicative Weights}

In the first scheme, weights decrease multiplicatively. For a given
parameter $0 < \gamma < 1$, weights of covered positive examples
decrease as follows: $w(e_j,i) = \gamma^i$, where $w(e_j,i)$ is the
weight of example $e_j$ being covered by $i$ rules. Note that the
weighted covering algorithm with $\gamma = 1$ would result in finding
the same rule over and over again, whereas with $\gamma = 0$ the
algorithm would perform the same as the standard CN2 covering
algorithm.

\subsubsection{Additive Weights}

In the second scheme, weights of covered positive examples decrease
according to the formula $w(e_j,i) = \frac{1}{i+1}$. In the first
iteration all target class examples contribute the same weight
$w(e_j,0) = 1$, while in the following iterations the contributions of
examples are inversely proportional to their coverage by previously
induced rules.

\subsection{Modified {\em \textit{WRAcc}} Heuristic with Example Weights}

The modification of CN2 reported in the work of \citet{tod:fla:lav:2000} affected
only the heuristic function: weighted relative accuracy was used as a
search heuristic, instead of the accuracy heuristic of the original
CN2, while everything else remained the same. In this work, the
heuristic function is further modified to handle example
weights, which provide the means to consider different parts of the
example space in each iteration of the weighted covering algorithm.

In the {\em \textit{WRAcc}} computation (Equation~\ref{eq:wracc}) all
probabilities are computed by relative frequencies. An example
weight measures how important it is to cover this example in the
next iteration. The modified {\em \textit{WRAcc}} measure is then defined
as follows:
%\begin{equation}
%\label{eq:modwracc}
$$
   {\it \textit{WRAcc}}(Class \leftarrow Cond) = \frac{n'(Cond)}{N'} \cdot
(\frac{n'(Class.Cond)}{n'(Cond)} - \frac{n'(Class)}{N'}).
$$
%\end{equation}
\noindent In this equation, $N'$ is the sum of the weights of all
examples, $n'(Cond)$ is the sum of the weights of all covered
examples, and $n'(Class.Cond)$ is the sum of the weights of all
correctly covered examples.
\par
To add a rule to the generated rule set, the rule with the maximum
{\em \textit{WRAcc}} measure is chosen out of those rules in the search space,
which are not yet present in the rule set produced so far (all rules
in the final rule set are thus distinct, without duplicates).

\subsection{Probabilistic Classification}

Each CN2 rule returns a class distribution in terms of the number
of examples covered, as distributed over classes. The CN2
algorithm uses class distribution in classifying unseen instances
only in the case of unordered rule sets, where rules are induced
separately for each class. In the case of ordered decision lists,
the first rule that fires provides the classification. In our
modified {\em CN2-SD} algorithm, also in the ordered case all applicable
rules are taken into account, hence probabilistic
classification is used in both classifiers. This means that the
terminology `ordered' and `unordered', which in CN2 distinguished
between decision list and rule set induction, has a different
meaning in our setting: the `unordered' algorithm refers to
learning classes one by one, while the `ordered' algorithm refers
to finding best rule conditions and assigning the majority class
in the rule head.

Note that {\em CN2-SD} does not use the same probabilistic
classification scheme as CN2. Unlike CN2, where the rule class
distribution is computed in terms of the numbers of examples covered,
{\em CN2-SD} treats the class distribution in terms of probabilities
(computed by the relative frequency estimate).
Table~\ref{tab-ruleset-birds-CN2-SD} presents the three rules of Table
\ref{tab-ruleset-birds} with the class distribution expressed with
probabilities.  A two-legged, feathered, large, non-flying animal with
a beak is again classified as a bird but now the probabilities are
averaged (instead of summing the numbers of examples), resulting in
the final probability distribution [0.72,0.28]. By using this voting
scheme the subgroups covering a small number of examples are not so
heavily penalized (as is the case
in CN2) when classifying a new example.

\begin{table}[t]
\begin{center}
\vspace*{0.5cm}
\begin{tabular}{lllllll}
{\tt if} & legs = 2   & \& & feathers = yes & {\tt then} & class = bird     & [1, 0] \\
{\tt if} & beak = yes &    &                & {\tt then} & class = bird     & [1, 0] \\
{\tt if} & size = large & \& & flies = no   & {\tt then} & class = elephant & [0.17,0.83]
\end{tabular}
\caption{The rule set of Table~\ref{tab-ruleset-birds} as treated by {\em CN2-SD}.
\label{tab-ruleset-birds-CN2-SD}}
\end{center}
\end{table}

\subsection{{\em CN2-SD} Implementation}

Two variants of {\em CN2-SD} have been implemented. The {\em CN2-SD} subgroup
discovery algorithm used in the experiments in this paper is
implemented in C and runs on a number of UNIX platforms. Its
predecessor, used in the experiments reported by 
\citet{Lavracetal.2002}, is implemented in Java and incorporated
in the WEKA data mining environment \citep{wit:fra:1999}. The C
implementation is more efficient and less restrictive than the Java
implementation, which is limited to binary class problems and to
discrete attributes.

\section{ROC Analysis for Subgroup Discovery}
\label{sec-ROC}

In this section we describe how ROC (Receiver Operating Characteristic)
analysis \citep{ProvostandFawcett2001} can be used to understand
subgroup discovery and to visualize and evaluate discovered subgroups.

A point in {\em ROC space} shows classifier performance in terms
of false alarm or {\em false positive rate} $ FPr = \frac{FP}{TN +
FP}$ (plotted on the $X$-axis), and sensitivity or {\em true
positive rate} $TPr = \frac{TP}{TP + FN} $ (plotted on the
$Y$-axis). In terms of the expressions introduced in
Sections~\ref{sec-2.1} and \ref{sec-2.2}, $TP$ (true positives),
$FP$ (false positives), $TN$ (true negatives) and $FN$ (false
negatives) can be expressed as: $TP=n(Class.Cond)$,
$FP=n(\overline{Class}.Cond)$,
$TN=n(\overline{Class}.\overline{Cond})$ and
$FN=n(Class.\overline{Cond})$, where $\overline{Class}$ and
$\overline{Cond}$ stand for $\neg {Class}$ and $\neg {Cond}$,
respectively.

The ROC space is appropriate for measuring the success of subgroup
discovery, since rules/sub\-groups whose $TPr/FPr$ tradeoff is close
to the diagonal can be discarded as insignificant. Conversely,
significant rules/sub\-groups are those sufficiently distant from the
diagonal. Significant rules define the points in ROC space from which
a convex hull can be constructed. The best rules define the ROC convex
hull. Figure~\ref{fig-groups-in-ROC} shows seven rules on the convex
hull (marked by {\small $\blacklozenge$}), while two rules $B1$ and
$B2$ below the convex hull (marked by x) are of lower quality.

\begin{figure}[t]
\begin{center}
\centering
\vspace*{0.5cm}
%\hspace{-1.5cm}
\includegraphics[width=2.75in]{jair2002_ROC-nada}
%\epsfxsize=2.75in \epsfbox{jair2002_ROC-nada.eps}
\caption{The ROC space with $TPr$ on the $X$ axis and $FPr$ on the $Y$
  axis. The solid line connecting seven optimal subgroups (marked by
  {\small $\blacklozenge$}) is the ROC convex hull. $B1$ and $B2$
  denote suboptimal subgroups (marked by x). The dotted line -- the
  diagonal connecting points (0,0) and (100,100) -- indicates positions
  of insignificant rules with zero relative accuracy.
\label{fig-groups-in-ROC}}
\end{center}
\end{figure}

\subsection{The Interpretation of Weighted Relative Accuracy in ROC Space}
\label{sec-ROC-WRAcc}

Weighted relative accuracy is appropriate for measuring the quality of
a single subgroup, because it is proportional to the distance from the
diagonal in ROC space.\footnote{Some of the reasoning supporting this
  claim is further discussed in the last two paragraphs of
  Section~\ref{sec-quality}.} To see that this holds, note first that
rule accuracy $p(Class|Cond)$ is proportional to the angle between the
$X$-axis and the line connecting the origin with the point depicting
the rule in terms of its $TPr/FPr$ tradeoff in ROC space. So, for
instance, the $X$-axis has always rule accuracy 0 (these are purely
negative subgroups), the $Y$-axis has always rule accuracy 1 (purely
positive subgroups), and the diagonal represents subgroups with rule
accuracy $p(Class)$, the prior probability of the positive class.
Consequently, all point on the diagonal represent insignificant
subgroups.

Relative accuracy, $p(Class|Cond) - p(Class)$, re-normalizes this such
that all points on the diagonal have relative accuracy 0, all points
on the $Y$-axis have relative accuracy
$1-p(Class)=p(\overline{Class})$ (the prior probability of the
negative class), and all points on the $X$-axis have relative accuracy
$-p(Class)$. Notice that all points on the diagonal also have
$\textit{WRAcc}=0$. In terms of subgroup discovery, the diagonal
represents all subgroups with the same target distribution as the
whole population; only the generality of these `average' subgroups
increases when moving from left to right along the diagonal. This
interpretation is slightly different in classifier learning, where the
diagonal represents random classifiers that can be constructed without
any training.

More generally, $\textit{WRAcc}$ isometrics lie on straight lines
parallel to the diagonal \citep{flach-icml03,furnkranz-flach-icml03}.
Consequently, a point on the line $TPr = FPr +a$, where $a$ is the
vertical distance of the line to the diagonal, has $\textit{WRAcc} =
a.p(Class) p(\overline{Class})$.  Thus, given a fixed class
distribution, $\textit{WRAcc}$ is proportional to the vertical
distance $a$ to the diagonal.  In fact, the quantity $TPr-FPr$ would
be an alternative quality measure for subgroups, with the additional
advantage that it allows for comparison of subgroups from populations
with different class distributions.


\subsection{Methods for Constructing ROC Curves and AUC Evaluation}
\label{sec-ROC-Area}

Subgroups obtained by CN2-SD can be evaluated in ROC space in
two different ways.

\subsubsection{AUC-Method-1}

The first method treats each rule as a separate subgroup which is
plotted in ROC space in terms of its true and false positive rates
($TPr$ and $FPr$). We then generate the convex hull of this set of
points, selecting the subgroups which perform optimally under a
particular range of operating characteristics. The area under this ROC
convex hull ({\em AUC}) indicates the combined quality of the optimal
subgroups, in the sense that it does evaluate whether a particular
subgroup has anything to add in the context of all the other
subgroups. However, this method does not take account of any overlap
between subgroups, and subgroups not on the convex hull are simply
ignored.

Figure~\ref{fig-ROC-1} presents two ROC curves, showing the
performance of CN2 and {\em CN2-SD} algorithms on the Australian UCI
data set.

\begin{figure}[t]
\begin{center}
\includegraphics[width=75mm]{ROC-1}
%\psfig{figure=ROC-1.eps,width=75mm}
\end{center}
\vspace*{0cm} \caption{Example ROC curves (AUC-Method-1) on the
   Australian UCI data set: the solid curve for the standard CN2
   classification rule learner, and the dotted curve for
   {\em CN2-SD}.}
\label{fig-ROC-1}
\end{figure}
%\vspace*{0.5cm}

\subsubsection{AUC-Method-2}

The second method employs the combined probabilistic classifications
of all subgroups, as indicated below. If we always choose the most
likely predicted class, this corresponds to setting a fixed threshold
0.5 on the positive probability (the probability of the target class):
if the positive probability is larger than this threshold we predict
positive, else negative. The ROC curve can be constructed by varying
this threshold from 1 (all predictions negative, corresponding to
(0,0) in ROC space) to 0 (all predictions positive, corresponding to
(1,1) in ROC space). This results in $n+1$ points in ROC space, where
$n$ is the total number of classified examples (test instances).
Equivalently, we can order all the classified examples by decreasing
predicted probability of being positive, and tracing the ROC curve by
starting in (0,0), stepping up when the example is actually positive
and stepping to the right when it is negative, until we reach
(1,1).\footnote{In the case of ties, we make the appropriate number of
  steps up and to the right at once, drawing a diagonal line segment.}
Each point on this curve corresponds to a classifier defined by a
possible probability threshold, as opposed to AUC-Method-1, where a
point on the ROC curve corresponds to one of the optimal subgroups.
The ROC curve depicts a set of classifiers, whereas the area under
this ROC curve indicates the combined quality of all subgroups (i.e.,
the quality of the entire rule set). This method can be used with a
test set or in cross-validation, but the resulting curve is not
necessarily convex.\footnote{A description of this method applied to
  decision tree induction can be found in the paper by \citet{Ferrietal.2002}.}

Figure~\ref{fig-ROC-2} presents two ROC curves, showing the
performance of the CN2 and {\em CN2-SD} algorithms on the
Australian UCI data set. It is apparent from this figure that CN2
is badly overfitting on this data set, because almost all of its
ROC curve is below the diagonal. This is because CN2 has learned
many overly specific rules, which bias the predicted
probabilities. These overly specific rules are visible in
Figure~\ref{fig-ROC-1} as points close to the origin.

\begin{figure}[t]
\begin{center}
\includegraphics[width=75mm]{ROC-2}
%\psfig{figure=ROC-2.eps,width=75mm}
\end{center}
\vspace*{0cm} \caption{Example ROC curves (AUC-Method-2) on the
   Australian UCI data set: the solid curve for the standard CN2
   classification rule learner, and the dotted curve for
   {\em CN2-SD}.}
\label{fig-ROC-2}
\end{figure}
%\vspace*{0.5cm}

\subsubsection{Comparison of the Two AUC Methods}

Which of the two methods is more appropriate for subgroup discovery is
open for debate. The second method seems more appropriate if the
discovered subgroups are intended to be applied also in the
predictive setting, as a rule set (a model) used for
classification. Its advantage is also that it is easier to apply
cross-validation. In the experimental evaluation in
Section~\ref{sec-experiments} we use AUC-Method-2 in the comparison of
the predictive performance of rule learners.

An argument in favor of using AUC-Method-1 for subgroup evaluation is
based on the observation that AUC-Method-1 suggests to eliminate, from
the induced set of subgroup descriptions, those rules which are not on
the ROC convex hull. This seems appropriate, as the `best' subgroups
according to the {\em \textit{WRAcc}} evaluation measure, are subgroups
most distant from the ROC diagonal. However, disjoint subgroups, either
on or close to the convex hull, should not be eliminated, as (due to
disjoint coverage and possibly different symbolic descriptions) they
may represent interesting subgroups, regardless of the fact that there
is another `better' subgroup on the ROC convex hull, with a similar
$TPr/FPr$ tradeoff.

Notice that the area under ROC curve (AUC-Method-1) cannot be used
as a predictive quality measure when comparing different subgroup
miners, because it does not take into account the overlapping
structure of subgroups. An argument against the use of this measure is
here elaborated through a simple example.\footnote{We are grateful to
the anonymous reviewer who provided this illustrative example.} Consider
for instance two subgroup mining results, of say 3 subgroups in each
resulting rule set. The first result set consists of three disjoint
subgroups of equal size that together cover all the examples of the
selected $Class$ value and have a 100\% accuracy. Thus these three
subgroups are a perfect classifier for the $Class$ value. In ROC
space the three subgroups collapse at the point (0,1/3). The second
result set consists of three equal subgroups (having a maximum
overlap: with different descriptions, but equal extensions),
also with a 100\% accuracy and covering one third of the class
examples. Clearly the first result is better, but the representation
of the results in ROC space (and the area under ROC curve) is the
same for both cases.

\section{Subgroup Evaluation Measures}
\label{sec-evaluation}

In this section we distinguish between {\em predictive} and {\em
  descriptive} evaluation measures, which is in-line with the
distinction of predictive induction and descriptive induction made in
Section~\ref{sec-intro}.  Descriptive measures are used to evaluate
the quality of individual rules (individual patterns). These quality
measures are the most appropriate for subgroup discovery, as the task
of subgroup discovery is to induce individual patterns of interest.
Predictive measures are used in addition to descriptive measures just
to show that the {\em CN2-SD} subgroup discovery mechanisms perform
well also in the predictive induction setting, where the goal is to
induce a classifier.


\subsection{Descriptive Measures of Rule Interestingness}
\label{sec-quality}

Descriptive measures of rule interestingness evaluate each individual
subgroup and are thus appropriate for evaluating the success of
subgroup discovery. The proposed quality measures compute the average
over the induced set of subgroup descriptions, which enables the
comparison between different algorithms.

\begin{description}
\item[Coverage.] The average coverage measures the percentage of
  examples covered on average by one rule of the induced rule set.
  Coverage of a single rule $ R_i $ is defined as
  $$Cov(R_i) = Cov(Class \leftarrow Cond_i) = p(Cond_i) = \frac{n(Cond_i)}{N}.$$
  The average coverage of a rule set is computed as
  $$COV=\frac{1}{n_R}\sum_{i=1}^{n_R}{Cov(R_i)}, $$
  \noindent where $n_R$ is the number of induced rules.
\item[Support.]
  For subgroup discovery it is interesting to compute the overall
  support (the target coverage) as the percentage of target
  examples (positives) covered by the rules, computed as the true
  positive rate for the union of subgroups. Support of a rule is
  defined as the frequency of correctly classified covered
  examples:
  $$Sup(R_i) = Sup(Class \leftarrow Cond_i) = p(Class.Cond_i) = \frac{n(Class.Cond_i)}{N}.$$
  The overall support of a rule set is computed as
$$ SUP = \frac{1}{N} \sum_{Class_j} n(Class_j \cdot \bigvee _{Class_j \leftarrow Cond_i} Cond_i), $$
    \noindent where the examples covered by several rules are counted
    only once (hence the disjunction of rule conditions of rules with
    the same $Class_j$ value in the rule head).
  \item[Size.]  Size is a measure of complexity (the syntactical
    complexity of induced rules). The rule set size is computed as the
    number of rules in the induced rule set (including the default
    rule):
    $$ SIZE = n_R. $$
    \noindent In addition to rule set size used in this paper,
    complexity could be measured also by the average number of
    rules/subgroups per class, and the average number of features per
    rule.
\item[Significance.]
  Average rule significance is computed in terms of the likelihood
  ratio of a rule, normalized with the likelihood ratio of the
  significance threshold (99\%); the average is computed over all
  the rules. Significance \citep[or {\em
evidence}, in the terminology of][]
  {Klosgen1996} indicates how significant is a finding, if
  measured by this statistical criterion. In the CN2 algorithm
  \citep{cla:nib:1989}, significance is measured in terms of the
  likelihood ratio statistic of a rule as follows:
  \begin{equation}
  \label{eq:sig}
  Sig(R_i) = Sig(Class \leftarrow Cond_i)
    = 2 \cdot \sum_j n(Class_j.Cond_i) \cdot \log \frac{n(Class_j.Cond_i)}{n(Class_j)}.
  \end{equation}
  \noindent where for each class $Class_j$, $n(Class_j.Cond_i)$ denotes
  the number of instances of $Class_j$ in the set where the rule body
  holds true, and $n(Class_j)$ is the number of $Class_j$ instances.
  Note that although for each generated subgroup description one class
  is selected as the target class, the significance criterion measures
  the distributional unusualness unbiased to any particular class --
  as such, it measures the significance of rule condition only.

The average significance of a rule set is computed as:
  $$ SIG = \frac{1}{n_R}\sum_{i=1}^{n_R}{Sig(R_i)}. $$

\item[Unusualness.]
  Average rule unusualness is
  computed as the average {\em \textit{WRAcc}} computed over all the
  rules:
  $$ WRACC = \frac{1}{n_R}\sum_{i=1}^{n_R}{\textit{WRAcc}(R_i)}. $$
  \noindent As discussed in Section~\ref{sec-ROC-WRAcc}, {\em
    \textit{WRAcc}} is appropriate for measuring the unusualness of
  separate subgroups, because it is proportional to the vertical
  distance from the diagonal in the ROC space (see the underlying
  reasoning in Section~\ref{sec-ROC-WRAcc}).


\end{description}

As {\em \textit{WRAcc}} is proportional to the distance to the
diagonal in ROC space, {\em \textit{WRAcc}} also reflects rule
significance -- the larger {\em \textit{WRAcc}}, the more
significant the rule, and vice versa. As both {\em \textit{WRAcc}}
and rule significance measure the distributional unusualness of a
subgroup, they are the most important quality measures for subgroup
discovery. However, while significance only measures distributional
unusualness, {\em \textit{WRAcc}} takes also rule coverage into
account, therefore we consider {\em unusualness} -- computed by the
average {\em \textit{WRAcc}} -- to be the most appropriate measure for
subgroup quality evaluation.

As pointed out in Section~\ref{sec-ROC-WRAcc}, the quantity $TPr-FPr$
could be an alternative quality measure for subgroups, with the
additional advantage that we can use it to compare subgroups from
populations with different class distributions. However, in this paper
we are only concerned with comparing subgroups from the same population,
and we prefer {\em \textit{WRAcc}} because of its `$p$-$g$'
(precision-generality) interpretation, which is particularly suitable
for subgroup discovery.

\subsection{Predictive Measures of Rule Set Classification Performance}

Predictive measures evaluate a rule set, interpreting a set of subgroup
descriptions as a predictive model. Despite the fact that optimizing
accuracy is not the intended goal of subgroup discovery algorithms,
these measures can be used in order to provide a comparison of
{\em CN2-SD} with standard classification rule learners.

\begin{description}
\item[Predictive accuracy.]
  The percentage of correctly predicted instances.
  For a binary classification problem, rule set accuracy is
  computed as follows:
  $$ ACC  = \frac{TP+TN}{TP+TN+FP+FN}.$$
  \noindent Note that ACC measures the accuracy of the whole
  rule set on both positive and negative examples, while rule accuracy
  (defined as $Acc(Class \leftarrow Cond) = p(Class|Cond)$) measures
  the accuracy of a single rule on positives only.
  %Note that ACC measures the accuracy of the whole
  %ruleset at once and not the average accuracy of individual
  %rules.
\item[Area under ROC curve.]
  The AUC-Method-2, described in Section 4.2, applicable to rule sets
  is selected as the evaluation measure. It interprets a rule set as
  a probabilistic model, given all the different probability
  thresholds as defined through the probabilistic classification
  of test instances.
\end{description}

\section{Experimental Evaluation}
\label{sec-experiments}

For subgroup discovery, expert evaluation of results is of ultimate
interest. Nevertheless, before applying the proposed approach to a
particular problem of interest, we wanted to verify our claims that
the mechanisms implemented in the {\em CN2-SD} algorithm are indeed
appropriate for subgroup discovery. For this purpose we tested it on
selected UCI data sets. In this paper we use the same data sets as
in the work of \citet{tod:fla:lav:2000}. We have applied {\em CN2-SD} also to a real
life problem of traffic accident analysis; these results were evaluated
also by the expert.

\subsection{The Experimental Setting}
\label{sec-setting}

To test the applicability of {\em CN2-SD} to the subgroup
discovery task, we compare its performance with the performance
of the standard CN2 classification rule learning algorithm \citep[referred
to as {\em CN2-standard}, and described in the work of][]{cla:bos:1991}
as well as with the CN2 algorithm using {\it \textit{WRAcc}}
 \citep[{\em CN2-WRAcc}, described by][]{tod:fla:lav:2000}.

In this comparative study all the parameters of the CN2 algorithm
are set to their default values (beam-size = 5,
significance-threshold = 99\%). The results of the {\em CN2-SD}
algorithm are  computed using both multiplicative weights
(with $\gamma$ = 0.5, 0.7, 0.9)\footnote{Results obtained with
$\gamma$ = 0.7 are presented in the tables of Appendix~A but
not in the main part of the paper.} and additive weights.

We estimate the performance of the algorithms using stratified 10-fold
cross-validation. The obtained estimates are presented in terms of
their average values and standard deviations.

Statistical significance of the difference in performance compared
to {\em CN2-standard} is tested using the paired t-test (exactly the
same folds are used in all comparisons) with significance level of $
95 \% $: bold font and $\uparrow$ to the right of a result in all the
tables means that the algorithm is significantly better than {\em
  CN2-standard} while $\downarrow$ means it is significantly worse.
The same paired t-test is used to compare the different versions of
our algorithm with {\em CN2-standard} over all the data sets.

%-------------------------------------------------------------------
\setlength{\tabcolsep}{4pt}
\begin{table}[t]
\begin{center}
\vspace*{0.5cm}
{\footnotesize
\begin{tabular}{rlrrrrrc}
\hline\noalign{\smallskip}

 & Data set & \#Att. & \#D.att. & \#C.att. & \#Class & \#Ex. & Maj. Class (\%) \\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
  1 & australian      & 14 &  8 &  6 &  2 &  690 &  56\\
  2 & breast-w        &  9 &  9 &  0 &  2 &  699 &  66\\
  3 & bridges-td      &  7 &  4 &  3 &  2 &  102 &  85\\
  4 & chess           & 36 & 36 &  0 &  2 & 3196 &  52\\
  5 & diabetes        &  8 &  0 &  8 &  2 &  768 &  65\\
  6 & echo            &  6 &  1 &  5 &  2 &  131 &  67\\
  7 & german          & 20 & 13 &  7 &  2 & 1000 &  70\\
  8 & heart           & 13 &  6 &  7 &  2 &  270 &  56\\
  9 & hepatitis       & 19 & 13 &  6 &  2 &  155 &  79\\
 10 & hypothyroid     & 25 & 18 &  7 &  2 & 3163 &  95\\
 11 & ionosphere      & 34 &  0 & 34 &  2 &  351 &  64\\
 12 & iris            &  4 &  0 &  4 &  3 &  150 &  33\\
 13 & mutagen         & 59 & 57 &  2 &  2 &  188 &  66\\
 14 & mutagen-f       & 57 & 57 &  0 &  2 &  188 &  66\\
 15 & tic-tac-toe     &  9 &  9 &  0 &  2 &  958 &  65\\
 16 & vote            & 16 & 16 &  0 &  2 &  435 &  61\\
\hline
 17 & balance         &  4 &  0 &  4 &  3 &  625 &  46\\
 18 & car             &  6 &  6 &  0 &  4 & 1728 &  70\\
 19 & glass           &  9 &  0 &  9 &  6 &  214 &  36\\
 20 & image           & 19 &  0 & 19 &  7 & 2310 &  14\\
 21 & soya            & 35 & 35 &  0 & 19 &  683 &  13\\
 22 & waveform        & 21 &  0 & 21 &  3 & 5000 &  34\\
 23 & wine            & 13 &  0 & 13 &  3 &  178 &  40\\
\hline
\end{tabular}}
\caption{Properties of the UCI data sets.}
\label{tab-datasets}
\end{center}
%\vspace*{-1cm}
\end{table}
\setlength{\tabcolsep}{1.4pt}

\subsection{Experiments on UCI Data Sets}
\label{sec-UCI}

We experimentally evaluate our approach on 23 data sets from the UCI
Repository of Machine Learning Databases \citep{mur:aha:1994}.
Table~\ref{tab-datasets} gives an overview of the selected data sets
in terms of the number of attributes (total, discrete, continuous),
the number of classes, the number of examples, and the percentage of
examples of the majority class. These data sets have been widely used
in other comparative studies \citep{tod:fla:lav:2000}. We have divided
the data sets in two groups (Table~\ref{tab-datasets}), those with two
classes (binary data sets 1--16) and those with more then two classes
(multi-class data sets 17--23). This distinction is made as ROC
analysis is applied only on binary data sets.\footnote{This is a
  simplification (as multi-class AUC could also be computed as the
  average of AUCs computed by comparing all pairs of classes
  \citep{HandandTill2001}) that still provides sufficient evidence
  to support the claims of this paper.}


\subsubsection{Results of the Unordered {\em CN2-SD}}

Tables~\ref{tab-results-Summary-Unordered-Descriptive} and
\ref{tab-results-Summary-Unordered-Predictive} present summary
results of the UCI experiments, while details can be found in
Tables~\ref{tab-results-Coverage-Unordered}--\ref{tab-results-AUC-Unordered}
in Appendix~A. For each performance measure, the summary table
shows the average value over all the data sets, the significance
of the results compared to {\em CN2-standard} ($p$-value),
win/loss/draw in terms of the number of data sets in which the
results are better/worse/equal compared with {\em CN2-standard},
as well as the number of significant wins and losses.

The analysis shows that if multiplicative weights are used, most
results improve with the increased value of the $\gamma$
parameter. As in most cases the best {\em CN2-SD} variants are {\em CN2-SD}
with $\gamma = 0.9$ and with additive weights, and as using
additive weighs is the simpler method, the additive weights
setting is recommended as default for experimental use.

The summary of results in terms of descriptive measures of
interestingness is as follows.
\begin{itemize}
\item In terms of the average coverage per rule {\em CN2-SD} produces
  rules with significantly higher coverage (the higher the coverage the better
  the rule) than both {\em CN2-WRAcc} and {\em CN2-standard}. The
  coverage is increased by increasing the $\gamma$ parameter and the
  best results are achieved by $\gamma = 0.9$ and by additive
  weights.
\item {\em CN2-SD} induces rule sets with significantly larger
  overall support than {\em CN2-standard} meaning that it covers a
  higher percentage of target examples (positives) thus leaving a
  smaller number of examples unclassified.\footnote{CN2
  handles the unclassified examples by classifying them using the
  default rule -- the rule predicting the majority class.}

%-------------------------------------------------------------------
\begin{table}[t]
\begin{center}
\vspace*{0.3cm}
{\footnotesize
\begin{tabular}{lcrrrrrl}
\hline\noalign{\smallskip}
Performance
          & Data & \multicolumn{1}{c}{\bf CN2}
          &        \multicolumn{1}{c}{\bf CN2}
          &        \multicolumn{1}{c}{\bf CN2-SD}
          &        \multicolumn{1}{c}{\bf CN2-SD}
          &        \multicolumn{1}{c}{\bf CN2-SD}
          &        \multicolumn{1}{c}{\bf Detailed} \\
Measure
          & Sets & \multicolumn{1}{c}{\bf standard}
          &        \multicolumn{1}{c}{\bf WRAcc}
          &        \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
          &        \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
          &        \multicolumn{1}{c}{\bf (add.)}
          &        \multicolumn{1}{c}{\bf Results} \\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
{\bf Coverage (COV)}                      & 23 & 0.131 $\pm$  0.14 &     0.311  $\pm$ 0.17  & 0.403 $\pm$ 0.23 &     0.450  $\pm$ 0.26  & {\bf 0.486} $\pm$ 0.30 & Table~\ref{tab-results-Coverage-Unordered} \\
$\bullet$ significance -- $p$ value        &    &                   & 0.000                  & 0.000            & 0.000                  & {\bf 0.000}            &  \\
$\bullet$ win/loss/draw &    &                   & 22/1/0         & 22/1/0     & 23/0/0           & {\bf 22/1/0}    &  \\
$\bullet$ sig.win/sig.loss &    &                   & 21/1          & 22/0    & 22/0          & {\bf 21/1}    &  \\\hline
{\bf Support (SUP)}                       & 23 &  0.84 $\pm$  0.03 &      0.85  $\pm$ 0.03  & 0.90  $\pm$ 0.06 & {\bf 0.92} $\pm$ 0.06  &       0.91  $\pm$ 0.06 & Table~\ref{tab-results-Support-Unordered} \\
$\bullet$ significance -- $p$ value        &    &                   & 0.637                  & 0.000            & {\bf 0.000}            & 0.001                  &  \\
$\bullet$ win/loss/draw &    &                   & 13/10/0           & 18/5/0    & {\bf 20/3/0}    & 16/7/0          &  \\
$\bullet$ sig.win/sig.loss &    &                   & 5/4          & 13/1    & {\bf 18/0}    & 13/1          &  \\\hline
{\bf Size (SIZE)}                         & 23 & 18.18 $\pm$ 21.77 & {\bf 6.15} $\pm$ 4.49  & 6.25  $\pm$ 4.42 &      6.49  $\pm$ 4.57  &       6.35  $\pm$ 4.58 & Table~\ref{tab-results-Size-Unordered} \\
$\bullet$ significance -- $p$ value        &    &                   & {\bf 0.006}            & 0.007            & 0.007                  & 0.007                  &  \\
$\bullet$ win/loss/draw &    &                   & {\bf 22/1/0}    & 22/1/0     & 20/3/0           & 23/0/0           &  \\
$\bullet$ sig.win/sig.loss &    &                   & {\bf 22/0}    & 21/0    & 19/2          & 18/0          &  \\\hline
{\bf Significance (SIG)}                  & 23 &  2.11 $\pm$  0.46 &      8.97  $\pm$ 4.66  & 15.57 $\pm$ 6.05 &     16.92  $\pm$ 8.90  & {\bf 18.47} $\pm$ 9.00 & Table~\ref{tab-results-Significance-Unordered} \\
$\bullet$ significance -- $p$ value        &    &                   & 0.000                  & 0.000            & 0.000                  & {\bf 0.000}            &  \\
$\bullet$ win/loss/draw  &    &                   & 22/1/0         & 23/0/0     & 22/1/0           & {\bf 23/0/0}    &  \\
$\bullet$ sig.win/sig.loss &    &                   & 21/0          & 23/0    & 21/0          & {\bf 23/0}    &  \\\hline
{\bf Unusualness (WRACC)}                 & 23 & 0.017 $\pm$  0.02 &     0.056  $\pm$ 0.05  & 0.079 $\pm$ 0.06 &     0.088  $\pm$ 0.06  & {\bf 0.092} $\pm$ 0.07 & Table~\ref{tab-results-Unusualness-Unordered} \\
$\bullet$ significance -- $p$ value        &    &                   & 0.001                  & 0.000            & 0.000                  & {\bf 0.000}            &  \\
$\bullet$ win/loss/draw  &    &                   & 20/1/2          & 22/1/0     & 22/1/0           & {\bf 22/1/0}    &  \\
$\bullet$ sig.win/sig.loss &    &                   & 19/1          & 21/1    & 21/1          & {\bf 21/1}    &  \\\hline
\end{tabular}
\caption{Summary of the experimental results on the UCI data sets
(descriptive evaluation measures) for different variants of the
unordered algorithm using 10-fold stratified cross-validation. The
best results are shown in boldface.}
\label{tab-results-Summary-Unordered-Descriptive}}
\end{center}
\end{table}
\setlength{\tabcolsep}{1.4pt}

\item {\em CN2-WRAcc} and {\em CN2-SD} induce rule sets that are
  significantly smaller than {\em CN2-standard} (smaller rule sets are
  better), while rule sets of {\em CN2-WRAcc} and {\em CN2-SD} are
  comparable, despite the fact that {\em CN2-SD} uses weights to `recycle'
  examples and thus produces overlapping rules.
\item {\em CN2-SD} induces significantly better rules in terms of rule
  significance (rules with higher significance are better) computed
  by the average likelihood ratio: while the ratios achieved by {\em
  CN2-standard} are already significant at the 99\% level, this is
  further pushed up by {\em CN2-SD} with maximum values achieved
  by additive weights. An interesting question, to be verified in
  further experiments, is whether the weighted
  versions of the CN2 algorithm improve the significance
  of the induced subgroups also in the case when CN2
  rules are induced without applying the significance test.
\item In terms of rule unusualness which is of ultimate interest
  to the subgroup discovery task, {\em CN2-SD} produces rules with
  significantly higher average weighted relative accuracy than {\em
    CN2-standard}. Like in the case of average coverage per rule the
  unusualness is increased by increasing the $\gamma$ parameter and
  the best results are achieved by $\gamma = 0.9$ and by additive
  weights. Note that the unusualness of a rule, computed by its
  {\em \textit{WRAcc}}, is a combination of rule accuracy, coverage
  and prior probability of the target class.
\end{itemize}

%-------------------------------------------------------------------
\begin{table}[t]
\begin{center}
\vspace*{0.3cm}
{\footnotesize
%\begin{small}
\begin{tabular}{lcrrrrrl}
\hline\noalign{\smallskip}
%\tiny
Performance
          & Data & \multicolumn{1}{c}{\bf CN2}
          &        \multicolumn{1}{c}{\bf CN2}
          &        \multicolumn{1}{c}{\bf CN2-SD}
%         &        \multicolumn{1}{c}{\bf CN2-SD}
          &        \multicolumn{1}{c}{\bf CN2-SD}
          &        \multicolumn{1}{c}{\bf CN2-SD}
          &        \multicolumn{1}{c}{\bf Detailed}\\
Measure
          & Sets & \multicolumn{1}{c}{\bf standard}
          &        \multicolumn{1}{c}{\bf WRAcc}
          &        \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
%         &        \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
          &        \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
          &        \multicolumn{1}{c}{\bf (add.)}
          &        \multicolumn{1}{c}{\bf Results} \\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
{\bf Accuracy (ACC)}                       & 23 & {\bf 81.61} $\pm$ 11.66 & 78.12 $\pm$ 16.28 &      80.92  $\pm$ 16.04 & 81.07 $\pm$ 15.78 & 79.36 $\pm$ 16.24 & Table~\ref{tab-results-Accuracy-Unordered} \\
$\bullet$ significance -- $p$ value        &    &                         & 0.150             & 0.771                   & 0.818             & 0.344             &  \\
$\bullet$ win/loss/draw &    &                         & 10/12/1      & 17/6/0             & 19/4/0      & 15/8/0       &  \\
$\bullet$ sig.win/sig.loss &    &                         & 3/5     & 9/4            & 10/4     & 7/4      &  \\\hline
{\bf AUC-Method-2 (AUC)}                  & 16 &      82.16  $\pm$ 16.81 & 84.37 $\pm$ 9.87  & {\bf 86.75} $\pm$ 8.95  & 86.39 $\pm$ 10.32 & 86.33 $\pm$ 8.60  & Table~\ref{tab-results-AUC-Unordered} \\
$\bullet$ significance -- $p$ value        &    &                         & 0.563             & {\bf 0.175}             & 0.236             & 0.236             &  \\
$\bullet$ win/loss/draw &    &                         & 6/9/1        & {\bf 10/6/0}      & 9/7/0        & 10/6/0       &  \\
$\bullet$ sig.win/sig.loss &    &                         & 5/5       & {\bf 6/2}      & 7/4       & 6/3      &  \\\hline
\end{tabular}
\caption{Summary of the experimental results on the UCI data sets
  (predictive evaluation measures) for different variants of the
  unordered algorithm using 10-fold stratified cross-validation. The
  best results are shown in boldface.}
\label{tab-results-Summary-Unordered-Predictive}
}
%\end{small}
\end{center}
%\vspace*{-1cm}
\end{table}
\setlength{\tabcolsep}{1.4pt}

In terms of predictive measures of classification performance
results can be summarized as follows.
\begin{itemize}
\item {\em CN2-SD} improves the accuracy in comparison with
  {\em CN2-WRAcc} and performs comparable to {\em CN2-standard}
  (the difference is insignificant). Notice however that while optimizing
  predictive accuracy is the ultimate goal of CN2, for {\em CN2-SD}
  the goal is to optimize the coverage/relative-accuracy tradeoff.
\item In the computation of area under ROC curve (AUC-Method-2) due
  to the restriction of this method to binary class data sets, only
  16 binary data sets are used in the comparisons. Notice that
  {\em CN2-SD} improves the area under ROC curve compared to {\em
    CN2-WRAcc} and compared to {\em CN2-standard}, but the differences
  are not significant. The area under ROC curve however seems not
  to be affected by the parameter $\gamma$ or by the weighting
  approach of {\em CN2-SD}.

\end{itemize}

AUC performance is also illustrated by means of the results on the
Australian UCI data set in Figures~\ref{fig-ROC-1} and \ref{fig-ROC-2}
of Section~\ref{sec-ROC-Area}. The solid lines in these graphs
indicate ROC curves obtained by {\em CN2-standard} while the dotted
lines represent ROC curves for {\em CN2-SD} with additive weights.


\subsubsection{Results of the Ordered {\em CN2-SD}}

For completeness, the results for different versions of the
ordered algorithm are summarized in
Tables~\ref{tab-results-Summary-Ordered-Descriptive} and
\ref{tab-results-Summary-Ordered-Predictive}, without giving the
results for individual data sets in Appendix~A.
%Appendix~\ref{app-ordered} contains these results.
In our view, the unordered {\em CN2-SD} algorithm is more
appropriate for subgroup discovery than the ordered variant,
as it induces a set of rules for each target class in turn.


%---------------------------------------------------------------------
\begin{table}[h]
\begin{center}
\vspace*{0.3cm}
{\footnotesize
%\begin{small}
\begin{tabular}{lcrrrrrl}
\hline\noalign{\smallskip}
%\tiny
Performance
          &   Data & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2-SD}
%         & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
%         &  \multicolumn{1}{c}{\bf Detailed}
\\
Measure
            & Sets & \multicolumn{1}{c}{\bf standard}
            & \multicolumn{1}{c}{\bf WRAcc}
            & \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
%           & \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
            & \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
            & \multicolumn{1}{c}{\bf (add.)}
%           & \multicolumn{1}{c}{\bf Results}
\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
{\bf Coverage (COV)}                       & 23 &       0.174 $\pm$  0.18 &       0.351  $\pm$ 0.18 &  0.439 $\pm$ 0.25  &       0.420  $\pm$ 0.23  &  {\bf 0.527} $\pm$ 0.32 \\
$\bullet$ significance -- $p$ value        &    &                         &       0.000             &  0.000             &       0.000              &  {\bf 0.000}            \\
$\bullet$ win/loss/draw  &    &                         &       21/2/0     &  23/0/0      &       23/0/0       &  {\bf 22/1/0}    \\
$\bullet$ sig.win/sig.loss &    &                         &       20/1     &  22/0     &       22/0      &  {\bf 22/1}    \\\hline
%& Table~\ref{tab-results-Coverage-Ordered} \\
{\bf Support (SUP) }                      & 23 &        0.85 $\pm$  0.03 &       0.85   $\pm$ 0.03 &  0.87  $\pm$ 0.05  & {\bf  0.91}  $\pm$ 0.05  &       0.90   $\pm$ 0.06 \\
$\bullet$ significance -- $p$ value        &    &                         &       0.694             &  0.026             & {\bf  0.000}             &       0.000             \\
$\bullet$ win/loss/draw &    &                         &       12/11/0      &  14/9/0      & {\bf  18/5/0}     &       19/4/0      \\
$\bullet$ sig.win/sig.loss &    &                         &       4/4     &  11/2     & {\bf  16/1}     &       14/0     \\\hline
{\bf Size (SIZE)}                          & 23 &       17.87 $\pm$ 28.10 &  {\bf 4.13}  $\pm$ 2.73 &  4.30  $\pm$ 2.58  &       4.61   $\pm$ 2.64  &       4.27   $\pm$ 2.79 \\
$\bullet$ significance -- $p$ value        &    &                         &  {\bf 0.025}            &  0.026             &       0.030              &       0.025             \\
$\bullet$ win/loss/draw  &    &                         &  {\bf 21/1/1}    &  21/2/0      &       20/3/0       &       21/2/0      \\
$\bullet$ sig.win/sig.loss &    &                         &  {\bf 21/0}    &  20/1     &       19/1      &       20/0     \\\hline
% & Table~\ref{tab-results-Size-Ordered} \\
{\bf Significance (SIG)}                   & 23 &        1.87 $\pm$  0.47 &       8.86   $\pm$ 4.81 &  12.70 $\pm$ 7.11  &       14.80  $\pm$ 8.31  &  {\bf 18.11} $\pm$ 9.84 \\
$\bullet$ significance -- $p$ value        &    &                         &       0.000             &  0.000             &       0.000              &  {\bf 0.000}            \\
$\bullet$ win/loss/draw  &    &                         &       22/1/0      &  22/1/0      &       23/0/0       &  {\bf 22/1/0}    \\
$\bullet$ sig.win/sig.loss &    &                         &       22/0     &  22/0     &       22/0      &  {\bf 21/0}    \\\hline
% & Table~\ref{tab-results-Significance-Ordered} \\
{\bf Unusualness (WRACC)}                 & 23 &       0.024 $\pm$  0.02 &       0.060  $\pm$ 0.05 &  0.080 $\pm$ 0.06  &       0.082  $\pm$ 0.06  &  {\bf 0.100} $\pm$ 0.07 \\
$\bullet$ significance -- $p$ value        &    &                         &       0.001             &  0.000             &       0.000              &  {\bf 0.000}            \\
$\bullet$ win/loss/draw  &    &                         &       18/5/0     &  21/2/0      &       21/2/0       &  {\bf 22/1/0}    \\
$\bullet$ sig.win/sig.loss &    &                         &       17/2     &  20/1     &       20/2      &  {\bf 21/1}    \\\hline
\end{tabular}
\caption{Summary of the experimental results on the UCI data sets
(descriptive evaluation measures) for different variants of the
ordered algorithm using 10-fold stratified cross-validation. The
best results are shown in boldface.}
\label{tab-results-Summary-Ordered-Descriptive}}
%\end{small}
\end{center}
%\vspace*{-1cm}
\end{table}
\setlength{\tabcolsep}{1.4pt}

%---------------------------------------------------------------------
\begin{table}[t]
\begin{center}
\vspace*{0.3cm}
{\footnotesize
%\begin{small}
\begin{tabular}{lcrrrrrl}
\hline\noalign{\smallskip}
%\tiny
Performance
          & Data & \multicolumn{1}{c}{\bf CN2}
          &        \multicolumn{1}{c}{\bf CN2}
          &        \multicolumn{1}{c}{\bf CN2-SD}
%         &        \multicolumn{1}{c}{\bf CN2-SD}
          &        \multicolumn{1}{c}{\bf CN2-SD}
          &        \multicolumn{1}{c}{\bf CN2-SD}
%         &        \multicolumn{1}{c}{\bf Detailed}
\\
Measure
            & Sets & \multicolumn{1}{c}{\bf standard}
            &        \multicolumn{1}{c}{\bf WRAcc}
            &        \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
%           &        \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
            &        \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
            &        \multicolumn{1}{c}{\bf (add.)}
%           &        \multicolumn{1}{c}{\bf Results}
\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
{\bf Accuracy (ACC)}                       & 23 & {\bf  83.00} $\pm$ 10.30  & 78.34 $\pm$ 16.52  & 79.50 $\pm$ 16.68 &       81.10  $\pm$ 16.53 & 80.79 $\pm$ 16.61 \\
$\bullet$ significance -- $p$ value        &    &                           & 0.155              & 0.286             &       0.556              & 0.494             \\
$\bullet$ win/loss/draw  &    &                           & 8/15/0        & 14/9/0       &       15/8/0       & 15/8/0       \\
$\bullet$ sig.win/sig.loss &    &                           & 3/6       & 15/4      &       8/4       & 7/3      \\\hline
% & Table~\ref{tab-results-Accuracy-Ordered} \\
{\bf AUC-Method-2 (AUC)}                   & 16 &       81.89  $\pm$ 10.07  & 82.28 $\pm$ 10.11  & 84.37 $\pm$ 9.19  & {\bf  84.70} $\pm$ 8.53  & 83.79 $\pm$ 9.64  \\
$\bullet$ significance -- $p$ value        &    &                           & 0.721              & 0.026             & {\bf  0.005}             & 0.049             \\
$\bullet$ win/loss/draw &    &                           & 9/6/1         & 10/6/0       & {\bf  12/4/0}      & 10/6/0       \\
$\bullet$ sig.win/sig.loss &    &                           & 6/5        & 6/3      & {\bf  8/4}      & 6/4      \\\hline
% & Table~\ref{tab-results-AUC-Ordered} \\
\end{tabular}
\caption{Summary of the experimental results on the UCI data sets
(predictive evaluation measures) for different variants of the
ordered algorithm using 10-fold stratified cross-validation. The
best results are shown in boldface.}
\label{tab-results-Summary-Ordered-Predictive}}
%\end{small}
\end{center}
%\vspace*{-1cm}
\end{table}
\setlength{\tabcolsep}{1.4pt}


\subsection{Experiments in Traffic Accident Data Analysis}
\label{sec-traffic}

We have evaluated the {\em CN2-SD} algorithm also on a traffic accident
data set. This is a large real-world database (1.5 GB) containing 21 years of
police traffic accident reports (1979--1999). The analysis of
this database is not straightforward because of the volume of the data,
the amounts of noise and missing data, and the fact that there is no
clearly defined data mining target. As described below,
some preprocessing was needed before running the subgroup
discovery experiments. Results of experiments were shown to the
domain expert whose comments are included.

\subsubsection{The Traffic Accident Data Set}

The traffic accident database contains data about traffic
accidents and the vehicles and casualties involved.  The data
is organized in three linked tables: the ACCIDENT table, the VEHICLE
table and the CASUALTY table.  The ACCIDENT table consists of the
records of all accidents that happened over the given period of time
(1979--1999), the VEHICLE table contains data about the vehicles
involved in those accidents, and the CASUALTY table contains data
about the casualties involved in the accidents. Consider the
following example: ``Two vehicles crashed in a traffic accident and
three people were seriously injured in the crash''. In terms of the
traffic data set this is recorded as one record in the ACCIDENT table,
two records in the VEHICLE table and three records in the CASUALTY
table. The three tables are described in more detail below.
\begin{itemize}
\item The ACCIDENT table contains one record for
  each accident. The 30 attributes describing an accident can be
  divided in three groups: date and time of the accident,
  description of the road where the accident has occurred,
  and conditions under which the accident has occurred (such as weather
  conditions, light and junction details). In the ACCIDENT table there
  are more than 5 million records.
\item The VEHICLE table contains one record for each vehicle involved
  in an accident from the ACCIDENT table.  There can be one or many
  vehicles involved in a single accident. The VEHICLE table attributes
  describe the type of the vehicle, maneuver and direction of the
  vehicle (from and to), vehicle location on the road, junction
  location at impact, sex and age of the driver, alcohol test results,
  damage resulting from the accident, and the object that vehicle hit
  on and off carriageway.  There are 24 attributes in the VEHICLE
  table which contains almost 9 million records.
\item The CASUALTY table contains records about casualties for each of
  the vehicles in the VEHICLE table. There can be one or more
  casualties per vehicle.  The CASUALTY table contains 16 attributes
  describing sex and age of casualty, type of casualty (e.g.,
  pedestrian, cyclist, car occupant etc.), severity of casualty, if
  casualty type is pedestrian, what were his/her characteristics
  (location, movement, direction).  This table contains almost 7
  million records.
\end{itemize}

\subsubsection{Data Preprocessing}

The large volume of data in the traffic data set makes it
practically impossible to run any data mining algorithm on the whole
set. Therefore we have taken samples of the data set and performed
the experiments on these samples.
We focused on the ACCIDENT table and examined only the
accidents that happened in 7 districts (called Police Force Codes, or PFCs) across the UK.\footnote{For the sake of anonymity, the code
  numbers 1 through 7 do not correspond to the PFCs 1 through 7 used
  for Police Force Codes in the actual traffic accident database.} The
7 PFCs were chosen by the domain expert and represent typical PFCs
from clusters of PFCs with the same accident dynamics, analyzed by 
\citet{Ljubic:2002}. In this way we obtained 7 data sets (one for each
PFC) with some hundred thousands of examples each. We further sampled
this data to obtain approximately 2500 examples per data set. The
sample percentages are listed in Table~\ref{tab-traffic-datasets}
together with the other characteristics of these 7 sampled data sets.



\setlength{\tabcolsep}{4pt}
\begin{table}[t]
\begin{center}
\vspace*{0.3cm}
{\footnotesize
\begin{tabular}{cccc}
\hline\noalign{\smallskip}

    & Number of & Percentage of & Class distribution (\%) \\
PFC & Examples  & Sampled Accidents & fatal / serious / slight \\

\noalign{\smallskip}
\hline
\noalign{\smallskip}
  1 & 2555 & 0.3 & 1.76 / 24.85 / 73.39 \\
  2 & 2523 & 1.9 & 2.53 / 30.87 / 66.60 \\
  3 & 2501 & 4.8 & 0.56 / 12.35 / 87.09 \\
  4 & 2499 & 1.9 & 2.16 / 27.21 / 70.63 \\
  5 & 2522 & 9.2 & 1.90 / 23.39 / 74.71 \\
  6 & 2548 & 2.0 & 1.41 / 13.69 / 84.90 \\
  7 & 2788 & 1.4 & 0.97 / 16.25 / 82.78 \\
\hline
\end{tabular}
\caption{Properties of the traffic data set.}
\label{tab-traffic-datasets}
}
\end{center}
%\vspace*{-1cm}
\end{table}
\setlength{\tabcolsep}{1.4pt}

Among the 26 attributes describing each of the 7 data sets we
chose the attribute `accident severity' to be the class attribute.
The task that we have addressed was therefore to find subgroups of
accidents of a certain severity (`slight', `serious' or `fatal')
and characterize them in terms of attributes describing the
accident, such as: `road class', `speed limit', `light condition',
etc.


\subsubsection{Results of Experiments}

We want to investigate if by running {\em CN2-SD} on the data sets,
described in Table~\ref{tab-traffic-datasets}, we are able to get some
rules that are typical and different for distinct PFCs.

We used the same methodology to perform the experiments as in the case
of the UCI data sets of Section~\ref{sec-UCI}. The only difference is
that here we do not perform the area under ROC curve analysis, because
the data sets are not two-class. The results presented in
%Tables~\ref{tab-traffic-Coverage-Unordered}--\ref{tab-traffic-Accuracy-Unordered}
Tables~\ref{tab-traffic-Coverage-Unordered}--\ref{tab-traffic-Unusualness-Unordered}
show the same advantages of {\em CN2-SD} over {\em CN2-WRAcc} and
{\em CN2-standard} as shown by the results of experiments on the UCI
data sets.%
\footnote{Like in the UCI case, only the results of the unordered
versions of the algorithm are presented here, although the
experiments were done with both unordered and ordered variants of
the algorithms.}
In particular, {\em CN2-SD} produces
substantially smaller rule sets, where individual rules have higher
coverage and significance.

It should be noticed that these data sets have a very unbalanced
class distribution (most accidents are `slight' and only few are
`fatal', see Table~\ref{tab-traffic-datasets}). In terms of rule set
accuracy, all algorithms achieved roughly default performance which
is obtained by always predicting the majority class. Since
classification was not the main interest of this experiment, we
omit the results.

%----------------------------------------------------------------------
%\begin{onecolumn}
%begin{table}[!ht]
\begin{table}[t]
\begin{center}
\vspace*{0.3cm}
{\footnotesize
%\begin{small}
\begin{tabular}{lrrrrrr}
\hline\noalign{\smallskip}
%\tiny
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD} \\
    \#    & \multicolumn{1}{c}{\bf standard}
          & \multicolumn{1}{c}{\bf WRAcc}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
          & \multicolumn{1}{c}{\bf (add.)} \\
          & \multicolumn{1}{c}{\em COV $\pm$ sd} &
            \multicolumn{1}{c}{\em COV $\pm$ sd} &
            \multicolumn{1}{c}{\em COV $\pm$ sd} &
            \multicolumn{1}{c}{\em COV $\pm$ sd} &
            \multicolumn{1}{c}{\em COV $\pm$ sd} &
            \multicolumn{1}{c}{\em COV $\pm$ sd} \\

\noalign{\smallskip}
\hline
\noalign{\smallskip}
  1 & 0.056 $\pm$ 0.01 & {\bf 0.108$\uparrow$} $\pm$ 0.00 & {\bf 0.111$\uparrow$} $\pm$ 0.03 & {\bf 0.111$\uparrow$} $\pm$ 0.03 & {\bf 0.123$\uparrow$} $\pm$ 0.03 & {\bf 0.110$\uparrow$} $\pm$ 0.03 \\
  2 & 0.050 $\pm$ 0.10 & {\bf 0.113$\uparrow$} $\pm$ 0.04 & {\bf 0.127$\uparrow$} $\pm$ 0.05 & {\bf 0.127$\uparrow$} $\pm$ 0.04 & {\bf 0.129$\uparrow$} $\pm$ 0.05 & {\bf 0.151$\uparrow$} $\pm$ 0.04 \\
  3 & 0.140 $\pm$ 0.03 & 0.118                 $\pm$ 0.03 & 0.126                 $\pm$ 0.02 & 0.119                 $\pm$ 0.02 & 0.118                 $\pm$ 0.01 & 0.154                 $\pm$ 0.02 \\
  4 & 0.052 $\pm$ 0.01 & {\bf 0.105$\uparrow$} $\pm$ 0.03 & {\bf 0.105$\uparrow$} $\pm$ 0.04 & {\bf 0.120$\uparrow$} $\pm$ 0.04 & {\bf 0.122$\uparrow$} $\pm$ 0.04 & {\bf 0.116$\uparrow$} $\pm$ 0.04 \\
  5 & 0.075 $\pm$ 0.08 & {\bf 0.108$\uparrow$} $\pm$ 0.04 & {\bf 0.115$\uparrow$} $\pm$ 0.06 & {\bf 0.121$\uparrow$} $\pm$ 0.05 & {\bf 0.110$\uparrow$} $\pm$ 0.05 & {\bf 0.127$\uparrow$} $\pm$ 0.04 \\
  6 & 0.078 $\pm$ 0.06 & {\bf 0.118$\uparrow$} $\pm$ 0.03 & {\bf 0.134$\uparrow$} $\pm$ 0.05 & {\bf 0.122$\uparrow$} $\pm$ 0.06 & {\bf 0.124$\uparrow$} $\pm$ 0.06 & {\bf 0.120$\uparrow$} $\pm$ 0.05 \\
  7 & 0.116 $\pm$ 0.08 & 0.110                 $\pm$ 0.11 & 0.118                 $\pm$ 0.14 & 0.124                 $\pm$ 0.13 & 0.122                 $\pm$ 0.13 & {\bf 0.143$\uparrow$} $\pm$ 0.12 \\\hline
{\bf Average} & 0.081 $\pm$ 0.03
              & 0.111 $\pm$ 0.01
              & 0.120 $\pm$ 0.01
              & 0.121 $\pm$ 0.00
              & 0.121 $\pm$ 0.01
              & {\bf 0.132} $\pm$ 0.02 \\
$\bullet$ significance -- $p$ value           &  & 0.047       & 0.021       & 0.023       & 0.029       & {\bf 0.003}       \\
$\bullet$ win/loss/draw &  & 5/2/0  & 6/1/0 & 6/1/0  & 6/1/0  & {\bf 7/0/0} \\
$\bullet$ sig.win/sig.loss &  & 5/0 & 5/0 & 5/0 & 5/0 & {\bf 6/0} \\\hline
\end{tabular}
\caption{Experimental results on the traffic accident data sets.
Average coverage per rule with standard deviation
({\em COV $\pm$ sd}) for different variants of the unordered
algorithm.}
\label{tab-traffic-Coverage-Unordered}
}
%\end{small}
\end{center}
%\vspace*{-1cm}
\end{table}
\setlength{\tabcolsep}{1.4pt}

%----------------------------------------------------------------------
%\begin{onecolumn}
%begin{table}[!ht]
\begin{table}[t]
\begin{center}
\vspace*{0.3cm}
{\footnotesize
%\begin{small}
\begin{tabular}{lrrrrrr}
\hline\noalign{\smallskip}
%\tiny
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD} \\
    \#    & \multicolumn{1}{c}{\bf standard}
          & \multicolumn{1}{c}{\bf WRAcc}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
          & \multicolumn{1}{c}{\bf (add.)} \\
          & \multicolumn{1}{c}{\em SUP $\pm$ sd} &
            \multicolumn{1}{c}{\em SUP $\pm$ sd} &
            \multicolumn{1}{c}{\em SUP $\pm$ sd} &
            \multicolumn{1}{c}{\em SUP $\pm$ sd} &
            \multicolumn{1}{c}{\em SUP $\pm$ sd} &
            \multicolumn{1}{c}{\em SUP $\pm$ sd} \\

\noalign{\smallskip}
\hline
\noalign{\smallskip}
  1 & 0.86 $\pm$ 0.03 & 0.89                 $\pm$ 0.02 & 0.83                   $\pm$ 0.06 & {\bf 0.93$\uparrow$} $\pm$ 0.04 & {\bf 0.96$\uparrow$} $\pm$ 0.02 & {\bf 0.95$\uparrow$} $\pm$ 0.03 \\
  2 & 0.84 $\pm$ 0.02 & 0.85                 $\pm$ 0.09 & 0.85                   $\pm$ 0.02 & {\bf 0.92$\uparrow$} $\pm$ 0.04 & {\bf 0.93$\uparrow$} $\pm$ 0.00 & 0.84                 $\pm$ 0.08 \\
  3 & 0.81 $\pm$ 0.06 & 0.82                 $\pm$ 0.04 & {\bf 0.93$\uparrow$}   $\pm$ 0.02 & {\bf 0.90$\uparrow$} $\pm$ 0.05 & {\bf 0.97$\uparrow$} $\pm$ 0.01 & 0.85                 $\pm$ 0.06 \\
  4 & 0.80 $\pm$ 0.04 & {\bf 0.87$\uparrow$} $\pm$ 0.05 & 0.82                   $\pm$ 0.05 & 0.83                 $\pm$ 0.00 & {\bf 0.91$\uparrow$} $\pm$ 0.03 & 0.81                 $\pm$ 0.10 \\
  5 & 0.87 $\pm$ 0.08 & 0.85                 $\pm$ 0.03 & 0.80$\downarrow$       $\pm$ 0.03 & 0.83                 $\pm$ 0.06 & {\bf 0.94$\uparrow$} $\pm$ 0.02 & 0.83                 $\pm$ 0.08 \\
  6 & 0.84 $\pm$ 0.06 & {\bf 0.88$\uparrow$} $\pm$ 0.07 & 0.81                   $\pm$ 0.09 & {\bf 0.91$\uparrow$} $\pm$ 0.06 & {\bf 0.88$\uparrow$} $\pm$ 0.07 & {\bf 0.98$\uparrow$} $\pm$ 0.01 \\
  7 & 0.81 $\pm$ 0.08 & 0.83                 $\pm$ 0.05 & {\bf 0.90$\uparrow$}   $\pm$ 0.01 & 0.81                 $\pm$ 0.01 & {\bf 0.95$\uparrow$} $\pm$ 0.02 & {\bf 0.99$\uparrow$} $\pm$ 0.00 \\\hline
{\bf Average} & 0.83 $\pm$ 0.03
              & 0.85 $\pm$ 0.02
              & 0.85 $\pm$ 0.05
              & 0.88 $\pm$ 0.05
              & {\bf 0.93} $\pm$ 0.03
              & 0.89 $\pm$ 0.08 \\
$\bullet$ significance -- $p$ value           &  & 0.056       & 0.548       & 0.053       & {\bf 0.001}       & 0.092       \\
$\bullet$ win/loss/draw &  & 6/1/0  & 4/3/0  & 6/1/0  & {\bf 7/0/0} & 6/1/0  \\
$\bullet$ sig.win/sig.loss &  & 2/0 & 2/1 & 4/0 & {\bf 7/0} & 3/0 \\\hline
\end{tabular}
\caption{Experimental results on the traffic accident data sets.
Overall support of rule sets with standard deviation
({\em SUP $\pm$ sd}) for different variants of the unordered
algorithm.}
\label{tab-traffic-Support-Unordered}
}
%\end{small}
\end{center}
%\vspace*{-1cm}
\end{table}
\setlength{\tabcolsep}{1.4pt}

%------------------------------------------------------------------------
%\begin{onecolumn}
%begin{table}[!ht]
\begin{table}[t]
\begin{center}
\vspace*{0.3cm}
{\footnotesize
%\begin{small}
\begin{tabular}{lrrrrrr}
\hline\noalign{\smallskip}
%\tiny
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD} \\
    \#    & \multicolumn{1}{c}{\bf standard}
          & \multicolumn{1}{c}{\bf WRAcc}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
          & \multicolumn{1}{c}{\bf (add.)} \\
          & \multicolumn{1}{c}{\em SIZE $\pm$ sd} &
            \multicolumn{1}{c}{\em SIZE $\pm$ sd} &
            \multicolumn{1}{c}{\em SIZE $\pm$ sd} &
            \multicolumn{1}{c}{\em SIZE $\pm$ sd} &
            \multicolumn{1}{c}{\em SIZE $\pm$ sd} &
            \multicolumn{1}{c}{\em SIZE $\pm$ sd} \\

\noalign{\smallskip}
\hline
\noalign{\smallskip}
  1 & 16.7 $\pm$ 0.60 & {\bf 9.3$\uparrow$} $\pm$ 0.99 & {\bf 10.0$\uparrow$} $\pm$ 0.51 & {\bf 10.6$\uparrow$}   $\pm$ 0.46 & {\bf 10.6$\uparrow$}   $\pm$ 0.73 & 9.5 {\bf $\uparrow$}   $\pm$ 0.25 \\
  2 & 18.7 $\pm$ 1.28 & {\bf 9.2$\uparrow$} $\pm$ 0.33 & {\bf 10.0$\uparrow$} $\pm$ 0.20 & {\bf 10.3$\uparrow$}   $\pm$ 0.21 & {\bf 10.3$\uparrow$}   $\pm$ 0.56 & {\bf 11.1$\uparrow$}   $\pm$ 0.23 \\
  3 & 7.0  $\pm$ 0.30 & 8.6                 $\pm$ 0.95 & 9.2                  $\pm$ 0.19 & 10.2$\downarrow$       $\pm$ 0.14 & 9.5                    $\pm$ 0.35 & 9.8 $\downarrow$       $\pm$ 0.19 \\
  4 & 18.0 $\pm$ 1.39 & {\bf 9.9$\uparrow$} $\pm$ 0.59 & {\bf 10.4$\uparrow$} $\pm$ 0.31 & {\bf 11.2$\uparrow$}   $\pm$ 0.64 & {\bf 11.2$\uparrow$}   $\pm$ 0.24 & {\bf 10.3$\uparrow$}   $\pm$ 0.56 \\
  5 & 12.8 $\pm$ 1.44 & {\bf 9.6$\uparrow$} $\pm$ 0.19 & {\bf 10.1$\uparrow$} $\pm$ 0.51 & 11.2                   $\pm$ 0.84 & 11.6                   $\pm$ 0.96 & 9.7 {\bf $\uparrow$}   $\pm$ 0.21 \\
  6 & 12.5 $\pm$ 0.31 & {\bf 8.5$\uparrow$} $\pm$ 0.35 & 9.3 {\bf $\uparrow$} $\pm$ 0.51 & 8.7 {\bf $\uparrow$}   $\pm$ 0.91 & 9.4 {\bf $\uparrow$}   $\pm$ 0.60 & 8.5 {\bf $\uparrow$}   $\pm$ 0.39 \\
  7 & 8.6  $\pm$ 1.41 & 9.3                 $\pm$ 0.41 & 9.9                  $\pm$ 0.90 & 10.8$\downarrow$       $\pm$ 0.73 & 11.1$\downarrow$       $\pm$ 0.13 & 10.4                   $\pm$ 0.59 \\\hline
{\bf Average} & 13.47 $\pm$ 4.57
              & {\bf  9.20} $\pm$ 0.50
              &  9.84 $\pm$ 0.44
              & 10.42 $\pm$ 0.86
              & 10.53 $\pm$ 0.84
              &  9.90 $\pm$ 0.80 \\
$\bullet$ significance -- $p$ value           &  & {\bf 0.040}       & 0.066       & 0.123       & 0.127       & 0.075       \\
$\bullet$ win/loss/draw  &  & {\bf 5/2/0} & 5/2/0  & 5/2/0  & 5/2/0  & 5/2/0  \\
$\bullet$ sig.win/sig.loss &  & {\bf 5/0} & 5/0 & 4/2 & 4/1 & 5/1 \\\hline
\end{tabular}
\caption{Experimental results on the traffic accident data sets.
Sizes of rule sets with standard deviation ({\em SIZE
    $\pm$ sd}) for different variants of the unordered algorithm.}
\label{tab-traffic-Size-Unordered}
}
%\end{small}
\end{center}
%\vspace*{-1cm}
\end{table}
\setlength{\tabcolsep}{1.4pt}

%-------------------------------------------------------------------------------------------------------------%\begin{onecolumn}
%begin{table}[!ht]
\begin{table}[t]
\begin{center}
\vspace*{0.3cm}
{\footnotesize
%\begin{small}
\begin{tabular}{lrrrrrr}
\hline\noalign{\smallskip}
%\tiny
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD} \\
    \#    & \multicolumn{1}{c}{\bf standard}
          & \multicolumn{1}{c}{\bf WRAcc}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
          & \multicolumn{1}{c}{\bf (add.)} \\
          & \multicolumn{1}{c}{\em SIG $\pm$ sd} &
            \multicolumn{1}{c}{\em SIG $\pm$ sd} &
            \multicolumn{1}{c}{\em SIG $\pm$ sd} &
            \multicolumn{1}{c}{\em SIG $\pm$ sd} &
            \multicolumn{1}{c}{\em SIG $\pm$ sd} &
            \multicolumn{1}{c}{\em SIG $\pm$ sd} \\

\noalign{\smallskip}
\hline
\noalign{\smallskip}
  1 & 1.9 $\pm$ 0.82 & {\bf 7.0$\uparrow$} $\pm$ 0.31 & {\bf 8.7$\uparrow$}  $\pm$ 0.41 & {\bf 9.7$\uparrow$}  $\pm$ 0.59 & {\bf 9.4$\uparrow$}  $\pm$ 0.30 & {\bf 9.6$\uparrow$}  $\pm$ 0.45 \\
  2 & 1.9 $\pm$ 0.34 & {\bf 6.2$\uparrow$} $\pm$ 0.25 & {\bf 9.9$\uparrow$}  $\pm$ 0.26 & {\bf 9.8$\uparrow$}  $\pm$ 0.20 & {\bf 9.5$\uparrow$}  $\pm$ 0.81 & {\bf 9.8$\uparrow$}  $\pm$ 0.36 \\
  3 & 1.3 $\pm$ 0.27 & {\bf 6.6$\uparrow$} $\pm$ 0.61 & {\bf 8.4$\uparrow$}  $\pm$ 0.52 & {\bf 9.2$\uparrow$}  $\pm$ 0.54 & {\bf 11.5$\uparrow$} $\pm$ 0.75 & {\bf 9.3$\uparrow$}  $\pm$ 0.17 \\
  4 & 1.6 $\pm$ 0.10 & {\bf 7.6$\uparrow$} $\pm$ 0.14 & {\bf 8.5$\uparrow$}  $\pm$ 0.79 & {\bf 11.0$\uparrow$} $\pm$ 0.84 & {\bf 9.4$\uparrow$}  $\pm$ 0.80 & {\bf 11.1$\uparrow$} $\pm$ 0.24 \\
  5 & 1.6 $\pm$ 0.75 & {\bf 6.0$\uparrow$} $\pm$ 0.23 & {\bf 10.6$\uparrow$} $\pm$ 0.70 & {\bf 9.6$\uparrow$}  $\pm$ 0.76 & {\bf 12.5$\uparrow$} $\pm$ 0.43 & {\bf 9.1$\uparrow$}  $\pm$ 0.74 \\
  6 & 1.5 $\pm$ 0.87 & {\bf 8.5$\uparrow$} $\pm$ 0.41 & {\bf 8.3$\uparrow$}  $\pm$ 0.54 & {\bf 9.8$\uparrow$}  $\pm$ 0.24 & {\bf 9.9$\uparrow$}  $\pm$ 0.51 & {\bf 12.5$\uparrow$} $\pm$ 0.35 \\
  7 & 1.7 $\pm$ 0.49 & {\bf 6.8$\uparrow$} $\pm$ 0.75 & {\bf 8.7$\uparrow$}  $\pm$ 0.20 & {\bf 9.9$\uparrow$}  $\pm$ 0.63 & {\bf 9.2$\uparrow$}  $\pm$ 0.73 & {\bf 9.7$\uparrow$}  $\pm$ 0.40 \\\hline
{\bf Average} &  1.64 $\pm$ 0.20
              &  6.95 $\pm$ 0.86
              &  9.01 $\pm$ 0.89
              &  9.85 $\pm$ 0.56
              & {\bf 10.20} $\pm$ 1.28
              & 10.16 $\pm$ 1.21 \\
$\bullet$ significance -- $p$ value           &  & 0.000       & 0.000       & 0.000       & {\bf 0.000}       & 0.000       \\
$\bullet$ win/loss/draw  &  & 7/0/0  & 7/0/0  & 7/0/0  & {\bf 7/0/0} & 7/0/0  \\
$\bullet$ sig.win/sig.loss &  & 7/0 & 7/0 & 7/0 & {\bf 7/0} & 7/0 \\\hline
\end{tabular}
\caption{Experimental results on the traffic accident data sets.
Average significance per rule with standard deviation ({\em SIG
    $\pm$ sd}) for different variants of the unordered algorithm.}
\label{tab-traffic-Significance-Unordered}
}
%\end{small}
\end{center}
%\vspace*{-1cm}
\end{table}
\setlength{\tabcolsep}{1.4pt}

%----------------------------------------------------------------------
%\begin{onecolumn}
%begin{table}[!ht]
\begin{table}[t]
\begin{center}
\vspace*{0.3cm}
{\footnotesize
%\begin{small}
\begin{tabular}{lrrrrrr}
\hline\noalign{\smallskip}
%\tiny
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD} \\
    \#    & \multicolumn{1}{c}{\bf standard}
          & \multicolumn{1}{c}{\bf WRAcc}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
          & \multicolumn{1}{c}{\bf (add.)} \\
          & \multicolumn{1}{c}{\em WRACC $\pm$ sd} &
            \multicolumn{1}{c}{\em WRACC $\pm$ sd} &
            \multicolumn{1}{c}{\em WRACC $\pm$ sd} &
            \multicolumn{1}{c}{\em WRACC $\pm$ sd} &
            \multicolumn{1}{c}{\em WRACC $\pm$ sd} &
            \multicolumn{1}{c}{\em WRACC $\pm$ sd} \\

\noalign{\smallskip}
\hline
\noalign{\smallskip}
  1 & 0.013 $\pm$ 0.02 & {\bf 0.025$\uparrow$} $\pm$ 0.05 & {\bf 0.025$\uparrow$} $\pm$ 0.10 & {\bf 0.026$\uparrow$} $\pm$ 0.02 & {\bf 0.028$\uparrow$} $\pm$ 0.03 & {\bf 0.025$\uparrow$} $\pm$ 0.09 \\
  2 & 0.009 $\pm$ 0.07 & {\bf 0.018$\uparrow$} $\pm$ 0.05 & {\bf 0.021$\uparrow$} $\pm$ 0.00 & {\bf 0.021$\uparrow$} $\pm$ 0.04 & {\bf 0.021$\uparrow$} $\pm$ 0.02 & {\bf 0.025$\uparrow$} $\pm$ 0.04 \\
  3 & 0.052 $\pm$ 0.01 & 0.043                 $\pm$ 0.00 & 0.046                 $\pm$ 0.07 & 0.043                 $\pm$ 0.03 & 0.043                 $\pm$ 0.05 & 0.056                 $\pm$ 0.02 \\
  4 & 0.010 $\pm$ 0.09 & {\bf 0.021$\uparrow$} $\pm$ 0.06 & {\bf 0.021$\uparrow$} $\pm$ 0.05 & {\bf 0.024$\uparrow$} $\pm$ 0.09 & {\bf 0.024$\uparrow$} $\pm$ 0.00 & {\bf 0.023$\uparrow$} $\pm$ 0.07 \\
  5 & 0.019 $\pm$ 0.04 & {\bf 0.026$\uparrow$} $\pm$ 0.06 & {\bf 0.027$\uparrow$} $\pm$ 0.07 & {\bf 0.029$\uparrow$} $\pm$ 0.08 & {\bf 0.027$\uparrow$} $\pm$ 0.01 & {\bf 0.030$\uparrow$} $\pm$ 0.07 \\
  6 & 0.027 $\pm$ 0.03 & {\bf 0.041$\uparrow$} $\pm$ 0.06 & {\bf 0.047$\uparrow$} $\pm$ 0.05 & {\bf 0.042$\uparrow$} $\pm$ 0.05 & {\bf 0.043$\uparrow$} $\pm$ 0.07 & {\bf 0.042$\uparrow$} $\pm$ 0.07 \\
  7 & 0.038 $\pm$ 0.03 & 0.035                 $\pm$ 0.01 & 0.038                 $\pm$ 0.04 & 0.040                 $\pm$ 0.00 & 0.039                 $\pm$ 0.08 & 0.046                 $\pm$ 0.04 \\\hline
{\bf Average} & 0.024 $\pm$ 0.02
              & 0.030 $\pm$ 0.01
              & 0.032 $\pm$ 0.01
              & 0.032 $\pm$ 0.01
              & 0.032 $\pm$ 0.01
              & {\bf 0.035} $\pm$ 0.01 \\
$\bullet$ significance -- $p$ value           &  & 0.096       & 0.042       & 0.041       & 0.048       & {\bf 0.000}       \\
$\bullet$ win/loss/draw  &  & 5/2/0  & 5/2/0  & 6/1/0  & 6/1/0  & {\bf 7/0/0} \\
$\bullet$ sig.win/sig.loss &  & 5/0 & 5/0 & 5/0 & 5/0 & {\bf 5/0} \\\hline
\end{tabular}
\caption{Experimental results on the traffic accident data sets.
Unusualness of rule sets with standard deviation
({\em WRACC $\pm$ sd}) for different variants of the unordered algorithm.}
\label{tab-traffic-Unusualness-Unordered}
}
%\end{small}
\end{center}
%\vspace*{-1cm}
\end{table}
\setlength{\tabcolsep}{1.4pt}


\subsubsection{Evaluation by the Domain Expert}

We have further examined the rules induced by the {\em CN2-SD}
algorithm (using additive weights). We focused on rules with high
coverage and rules that cover a high percentage of the predicted
class as these are the rules that are likely to reflect some
regularity in the data.

One of the most interesting results concerned the following.  One
might expect that the number of people injured would increase with the
severity of the accident (up to the total number of occupants in the
vehicles). Furthermore, common sense would dictate that the number of
vehicles involved would also increase with accident severity.
Contrary to these expectations we found rules of the following two
kinds:

\begin{itemize}
\item Rules that cover more than the average proportion of `fatal' or
  `serious' accidents when just one vehicle is involved in the
  accident.  Examples of such rules are:\\
\begin{tt}
IF nv < 1.500 THEN sev = "1" [15 280 1024]\footnote{The rules
in the example are given in the {\em CN2-SD} output format where
{\tt nv} stands for `number of vehicles', {\tt nc} is the
`number of casualties' and {\tt "1"}, {\tt "2"}, and {\tt "3"}
denote the class values `fatal', `serious' and `slight'
respectively.} \\
IF nv < 1.500 THEN sev = "2" [22 252 890]
\end{tt}
\item Rules that cover more than the average proportion of `slight' accidents when
  two or more vehicles are involved and there are few casualties.
  An example of such a rule is:\\
\begin{tt}
IF nv > 1.500 AND nc < 2.500 THEN sev = "3" [8 140 1190]
\end{tt}
\end{itemize}

Having shown the induced results to the domain expert, he pointed out
the following aspects of data collection for the data in the
ACCIDENT table.\footnote{We have also shown the {\em CN2-standard} and {\em
    CN2-WRAcc} results to the expert but he did not consider any of
  the rules to be interesting.}
\begin{itemize}
\item The severity code in the ACCIDENT table relates to the most
  severe injury among those reported for that accident. Therefore a
  multiple vehicle accident with 1 fatal and 20 slight injuries would
  be classified as fatal as one fatality occurred, while each
  individual casualty injury severity is coded in the CASUALTY
  table.
\item Some (slight) injuries may be unreported at the accident scene:
  if the policeman compiled/revised the report after the event, new
  casualty/injury details can be reported (injuries that came to light
  after the event or reported for reasons relating to injury/insurance
  claims). However, these changes are not reflected in the ACCIDENT
  table.
\end{itemize}
The findings revealed by the rules were surprising to the domain expert
and need further investigation. The analysis shows that examining the
ACCIDENT table is not sufficient and that further examination of the
VEHICLE and CASUALTY tables is needed in further work.


\section{Related Work}
\label{sec-related}

Other systems have addressed the task of subgroup discovery, the best
known being EXPLORA \citep{Klosgen1996} and MIDOS
\citep{Wrobel1997,Wrobel2001}. EXPLORA treats the learning task as a
single relation problem, i.e., all the data are assumed to be
available in one table (relation), whereas MIDOS extends this task to
multi-relational databases. Other approaches deal with
multi-relational databases using propositionalisation and aggregate
functions can be found in the work of \citet{Knobbeetal.2001,Knobbeetal.2002}.

Another approach to finding symbolic descriptions of groups of
instances is symbolic clustering, which has been popular for many
years \citep{Michalski1980,GowdaandDiday1992}. Moreover, learning of
concept hierarchies also aims at discovering groups of instances,
which can be induced in a supervised or unsupervised manner: decision
tree induction algorithms perform supervised symbolic learning of
concept hierarchies \citep{Langley1996,DeRaedtandBlockeel1997}, whereas
hierarchical clustering algorithms
\citep{SokalandSneath1963,Gordon1982} are unsupervised and do not
result in symbolic descriptions. Note that in decision tree learning,
the rules which can be formed from paths leading from the root node to
class labels in the leaves represent {\em discriminant descriptions},
formed from properties that best discriminate between the classes.  As
rules formed from decision tree paths form discriminant descriptions,
they are inappropriate for solving subgroup discovery tasks which aim
at describing subgroups by their characteristic properties.

Instance weights play an important role in boosting
\citep{FreundandShapire1996} and alternating decision trees
\citep{SchapireandSinger1998}. Instance weights have been used also in
variants of the covering algorithm implemented in rule learning
approaches such as SLIPPER \citep{cohen1999}, RL \citep{Leeetal.1998}
and DAIRY \citep{Hsu1998}. A variant of the weighted covering algorithm
has been used in the subgroup discovery algorithm SD for rule subset
selection \citep{GambergerLavrac2002}.

A variety of rule evaluation measures and heuristics have been studied for
subgroup discovery \citep{Klosgen1996,Wrobel1997,Wrobel2001}, aimed at
balancing the size of a group (referred to as factor $g$) with its
distributional unusualness (referred to as factor $p$).  The
properties of functions that combine these two factors have been
extensively studied \citep[the so-called `$p$-$g$-space'][]{Klosgen1996}. An alternative measure $q=\frac{TP}{FP+par}$ was
proposed in the SD algorithm for expert-guided subgroup discovery
\citep{GambergerLavrac2002}, aimed at minimizing the number of false
positives $FP$, and maximizing true positives $TP$, balanced by
generalization parameter $par$. Besides such `objective' measures of
interestingness, some `subjective' measure of interestingness of a
discovered pattern can be taken into account, such as actionability
(`a pattern is interesting if the user can do something with it to
his or her advantage') and unexpectedness (`a pattern is interesting
to the user if it is surprising to the user') \citep{Silberschatz95}.

Note that some approaches to association rule induction can also be
used for subgroup discovery. For instance, the APRIORI-C algorithm
\citep{JovanoskiandLavrac2001}, which applies association rule
induction to classification rule induction, outputs classification
rules with guaranteed support and confidence with respect to a target
class.  If a rule satisfies also a user-defined significance
threshold, an induced APRIORI-C rule can be viewed as an independent
`chunk' of knowledge about the target class (selected property of
interest for subgroup discovery), which can be viewed as a subgroup
description with guaranteed significance, support and confidence. This
observation led to the development of a novel subgroup discovery
algorithm APRIORI-SD \citep{Kavseketal.2003}.

It should be noticed that in the terminology `patient vs. greedy' of
\citet{FriedmanandFisher1999}, {\em WRAcc} is a `patient' rule quality
measure, favoring more general subgroups than those found by using
`greedy' quality measures. As shown by our experiments in
\citet{tod:fla:lav:2000}, {\em WRAcc} heuristic improves rule coverage
compared to the standard CN2 heuristic. This observation is confirmed
also in the experimental evaluation in Section~\ref{sec-experiments}
of this paper.  Further evidence confirming this claim is provided by 
\citet{Kavseketal.2003}, providing experimental comparison of results
of {\em CN2-SD} and our novel subgroup discovery algorithm APRIORI-SD
with rule learners CN2, RIPPER and APRIORI-C.

\section{Conclusions and Further Work}
\label{sec-conclusions}

We have presented a novel approach to adapting standard
classification rule learning to subgroup discovery. To this end
we have appropriately adapted the covering algorithm, the search
heuristic, the probabilistic classification and the area under the
ROC curve (AUC) performance measure. We have also proposed a set
of metrics appropriate for evaluating the quality of induced
subgroup descriptions.

The experimental results on 23 UCI data sets demonstrate that
{\em CN2-SD} produces substantially smaller rule sets, where
individual rules have higher coverage and significance. These
three factors are important for subgroup discovery: smaller
size enables better understanding, higher coverage means
larger support, and higher significance means that rules
describe discovered subgroups that are significantly different
from the entire population. We have evaluated the results of
{\em CN2-SD} also in terms of AUC and shown a small
(insignificant) increase in terms of the area under ROC curve.

We have applied {\em CN2-SD} also to a real-life problem of traffic
accident analysis. The experimental results confirm the findings in
the UCI data sets. The most interesting findings are due to
interpretation by the domain expert. What was confirmed in this case
study was that the result of a data mining process depends not only on
the appropriateness of the selected method and the data that is at
hand but also on how the data has been collected. In the traffic
accident experiments examining the ACCIDENT table was not sufficient,
and further examination of the VEHICLE and CASUALTY tables is needed.
This will be performed using the RSD relational subgroup discovery
algorithm \citep{Lavracetal.2002a}, a recent upgrade of the {\em
  CN2-SD} algorithm which enables relational subgroup discovery.

In further work we plan to compare the results with the MIDOS
subgroup discovery algorithm.  We plan to investigate the behavior
of {\em CN2-SD} in terms of AUC in multi-class
problems~\citep{HandandTill2001}. An interesting question, to be
verified in further experiments, is whether the weighted versions
of the CN2 algorithm improve the significance of the induced
subgroups also in the case when CN2 rules are induced without
applying the significance test.

An important aspect of subgroup discovery performance, which is
neglected in our study, is the degree of overlap of the induced
subgroups. The challenge of our further research is to propose
extensions of the weighted relative accuracy heuristic and ROC
space evaluation metrics that will take into account the overlap
of subgroups.

We are now moving the focus of our research in subgroup discovery
from heuristic search toward exhaustive search of the space of
patterns. An attempt of this kind is described by 
\citet{Kavseketal.2003} where the well known APRIORI association rule
learner was adapted to the task of subgroup discovery.

\acks{Thanks to Dragan Gamberger for joint work on the weighted
covering algorithm, and Jos\'e Hern\'an\-dez-Orallo and Cesar
Ferri-Ram\'{\i}rez for joint work on AUC. Thanks to Peter
Ljubi\v{c} and Damjan Dem\v{s}ar for the help in upgrading the C
code of the original CN2 algorithm. We are grateful to John Bullas
for the evaluation of the results of traffic accident data
analysis. Thanks are also due to the anonymous reviewers for their
insightful comments. The work reported in this paper was supported by the
Slovenian Ministry of Education, Science and Sport, the
IST-1999-11495 project Data Mining and Decision Support for
Business Competitiveness: A European Virtual Enterprise, and the
British Council project Partnership in Science PSP-18.}

%\newpage
\appendix
\section{Tables with Detailed Results for Different
Variants of the Unordered Algorithm in UCI Data Sets}
\label{app-unordered}
%\vspace*{1cm}

The tables in this appendix show detailed results of the
performance of different variants of the unordered algorithm.  The
comparisons are made on 23 UCI data sets listed in
Table~\ref{tab-datasets}. The results shown in
Tables~\ref{tab-results-Coverage-Unordered}--\ref{tab-results-Unusualness-Unordered}
of Appendix~A are summarized in the paper in
Table~\ref{tab-results-Summary-Unordered-Descriptive}, and the
results of
Tables~\ref{tab-results-Accuracy-Unordered}--\ref{tab-results-AUC-Unordered}
in Table~\ref{tab-results-Summary-Unordered-Predictive}.

%-----------------------------------------------------------------------
%\begin{onecolumn}
%begin{table}[!ht]
\begin{table}[!h]
\begin{center}
\vspace*{0.3cm} {\footnotesize
%\begin{small}
\begin{tabular}{lrrrrrr}
\hline\noalign{\smallskip}
%\tiny
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD} \\
\#        & \multicolumn{1}{c}{\bf standard}
          & \multicolumn{1}{c}{\bf WRAcc}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
          & \multicolumn{1}{c}{\bf (add.)} \\
          & \multicolumn{1}{c}{\em COV $\pm$ sd} &
            \multicolumn{1}{c}{\em COV $\pm$ sd} &
            \multicolumn{1}{c}{\em COV $\pm$ sd} &
            \multicolumn{1}{c}{\em COV $\pm$ sd} &
            \multicolumn{1}{c}{\em COV $\pm$ sd} &
            \multicolumn{1}{c}{\em COV $\pm$ sd} \\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
  1 & 0.071 $\pm$ 0.01 & {\bf 0.416$\uparrow$} $\pm$ 0.00 & {\bf 0.473$\uparrow$} $\pm$ 0.03 & {\bf 0.492$\uparrow$} $\pm$ 0.03 & {\bf 0.480$\uparrow$} $\pm$ 0.03 & {\bf 0.424$\uparrow$} $\pm$ 0.03 \\
  2 & 0.079 $\pm$ 0.10 & {\bf 0.150$\uparrow$} $\pm$ 0.04 & {\bf 0.208$\uparrow$} $\pm$ 0.05 & {\bf 0.174$\uparrow$} $\pm$ 0.04 & {\bf 0.218$\uparrow$} $\pm$ 0.05 & {\bf 0.260$\uparrow$} $\pm$ 0.04 \\
  3 & 0.625 $\pm$ 0.03 & 0.322$\downarrow$     $\pm$ 0.03 & 0.612                 $\pm$ 0.02 & 0.617                 $\pm$ 0.02 & 0.721                 $\pm$ 0.01 & 0.330$\downarrow$     $\pm$ 0.02 \\
  4 & 0.048 $\pm$ 0.01 & {\bf 0.496$\uparrow$} $\pm$ 0.03 & {\bf 0.504$\uparrow$} $\pm$ 0.04 & {\bf 0.513$\uparrow$} $\pm$ 0.04 & {\bf 0.504$\uparrow$} $\pm$ 0.04 & {\bf 0.507$\uparrow$} $\pm$ 0.04 \\
  5 & 0.057 $\pm$ 0.08 & {\bf 0.275$\uparrow$} $\pm$ 0.04 & {\bf 0.296$\uparrow$} $\pm$ 0.06 & {\bf 0.344$\uparrow$} $\pm$ 0.05 & {\bf 0.299$\uparrow$} $\pm$ 0.05 & {\bf 0.381$\uparrow$} $\pm$ 0.04 \\
  6 & 0.312 $\pm$ 0.06 & {\bf 0.576$\uparrow$} $\pm$ 0.03 & {\bf 0.936$\uparrow$} $\pm$ 0.05 & {\bf 1.039$\uparrow$} $\pm$ 0.06 & {\bf 1.006$\uparrow$} $\pm$ 0.06 & {\bf 1.295$\uparrow$} $\pm$ 0.05 \\
  7 & 0.053 $\pm$ 0.08 & {\bf 0.092$\uparrow$} $\pm$ 0.11 & {\bf 0.141$\uparrow$} $\pm$ 0.14 & {\bf 0.153$\uparrow$} $\pm$ 0.13 & {\bf 0.138$\uparrow$} $\pm$ 0.13 & {\bf 0.151$\uparrow$} $\pm$ 0.12 \\
  8 & 0.107 $\pm$ 0.09 & {\bf 0.240$\uparrow$} $\pm$ 0.07 & {\bf 0.419$\uparrow$} $\pm$ 0.09 & {\bf 0.376$\uparrow$} $\pm$ 0.12 & {\bf 0.366$\uparrow$} $\pm$ 0.11 & {\bf 0.435$\uparrow$} $\pm$ 0.09 \\
  9 & 0.207 $\pm$ 0.04 & {\bf 0.430$\uparrow$} $\pm$ 0.06 & {\bf 0.637$\uparrow$} $\pm$ 0.04 & {\bf 0.829$\uparrow$} $\pm$ 0.04 & {\bf 0.826$\uparrow$} $\pm$ 0.04 & {\bf 0.686$\uparrow$} $\pm$ 0.03 \\
 10 & 0.093 $\pm$ 0.00 & {\bf 0.495$\uparrow$} $\pm$ 0.00 & {\bf 0.509$\uparrow$} $\pm$ 0.00 & {\bf 0.509$\uparrow$} $\pm$ 0.00 & {\bf 0.516$\uparrow$} $\pm$ 0.00 & {\bf 0.513$\uparrow$} $\pm$ 0.00 \\
 11 & 0.099 $\pm$ 0.05 & {\bf 0.168$\uparrow$} $\pm$ 0.08 & {\bf 0.229$\uparrow$} $\pm$ 0.05 & {\bf 0.234$\uparrow$} $\pm$ 0.04 & {\bf 0.246$\uparrow$} $\pm$ 0.04 & {\bf 0.354$\uparrow$} $\pm$ 0.06 \\
 12 & 0.378 $\pm$ 0.01 & 0.386                 $\pm$ 0.01 & {\bf 0.619$\uparrow$} $\pm$ 0.00 & 0.444                 $\pm$ 0.00 & {\bf 0.768$\uparrow$} $\pm$ 0.00 & {\bf 0.668$\uparrow$} $\pm$ 0.01 \\
 13 & 0.160 $\pm$ 0.11 & {\bf 0.408$\uparrow$} $\pm$ 0.09 & {\bf 0.639$\uparrow$} $\pm$ 0.15 & {\bf 0.467$\uparrow$} $\pm$ 0.16 & {\bf 0.424$\uparrow$} $\pm$ 0.18 & {\bf 0.621$\uparrow$} $\pm$ 0.17 \\
 14 & 0.142 $\pm$ 0.01 & {\bf 0.356$\uparrow$} $\pm$ 0.07 & {\bf 0.461$\uparrow$} $\pm$ 0.02 & {\bf 0.668$\uparrow$} $\pm$ 0.03 & {\bf 0.569$\uparrow$} $\pm$ 0.03 & {\bf 0.720$\uparrow$} $\pm$ 0.03 \\
 15 & 0.030 $\pm$ 0.01 & {\bf 0.113$\uparrow$} $\pm$ 0.07 & {\bf 0.129$\uparrow$} $\pm$ 0.02 & {\bf 0.146$\uparrow$} $\pm$ 0.03 & {\bf 0.182$\uparrow$} $\pm$ 0.03 & {\bf 0.117$\uparrow$} $\pm$ 0.03 \\
 16 & 0.129 $\pm$ 0.01 & {\bf 0.650$\uparrow$} $\pm$ 0.07 & {\bf 0.703$\uparrow$} $\pm$ 0.02 & {\bf 0.711$\uparrow$} $\pm$ 0.03 & {\bf 0.674$\uparrow$} $\pm$ 0.03 & {\bf 0.831$\uparrow$} $\pm$ 0.03 \\\hline
 17 & 0.021 $\pm$ 0.00 & {\bf 0.216$\uparrow$} $\pm$ 0.00 & {\bf 0.225$\uparrow$} $\pm$ 0.00 & {\bf 0.270$\uparrow$} $\pm$ 0.00 & {\bf 0.307$\uparrow$} $\pm$ 0.00 & {\bf 0.324$\uparrow$} $\pm$ 0.00 \\
 18 & 0.022 $\pm$ 0.05 & {\bf 0.146$\uparrow$} $\pm$ 0.08 & {\bf 0.155$\uparrow$} $\pm$ 0.05 & {\bf 0.157$\uparrow$} $\pm$ 0.04 & {\bf 0.166$\uparrow$} $\pm$ 0.04 & {\bf 0.200$\uparrow$} $\pm$ 0.06 \\
 19 & 0.066 $\pm$ 0.01 & {\bf 0.331$\uparrow$} $\pm$ 0.01 & {\bf 0.357$\uparrow$} $\pm$ 0.00 & {\bf 0.628$\uparrow$} $\pm$ 0.00 & {\bf 0.616$\uparrow$} $\pm$ 0.00 & {\bf 0.759$\uparrow$} $\pm$ 0.01 \\
 20 & 0.039 $\pm$ 0.11 & {\bf 0.139$\uparrow$} $\pm$ 0.09 & {\bf 0.151$\uparrow$} $\pm$ 0.15 & {\bf 0.159$\uparrow$} $\pm$ 0.16 & {\bf 0.149$\uparrow$} $\pm$ 0.18 & {\bf 0.169$\uparrow$} $\pm$ 0.17 \\
 21 & 0.040 $\pm$ 0.01 & {\bf 0.076$\uparrow$} $\pm$ 0.07 & {\bf 0.115$\uparrow$} $\pm$ 0.02 & {\bf 0.177$\uparrow$} $\pm$ 0.03 & {\bf 0.172$\uparrow$} $\pm$ 0.03 & {\bf 0.216$\uparrow$} $\pm$ 0.03 \\
 22 & 0.004 $\pm$ 0.01 & {\bf 0.185$\uparrow$} $\pm$ 0.07 & {\bf 0.194$\uparrow$} $\pm$ 0.02 & {\bf 0.185$\uparrow$} $\pm$ 0.03 & {\bf 0.188$\uparrow$} $\pm$ 0.03 & {\bf 0.191$\uparrow$} $\pm$ 0.03 \\
 23 & 0.231 $\pm$ 0.01 & {\bf 0.477$\uparrow$} $\pm$ 0.07 & {\bf 0.552$\uparrow$} $\pm$ 0.02 & {\bf 0.715$\uparrow$} $\pm$ 0.03 & {\bf 0.818$\uparrow$} $\pm$ 0.03 & {\bf 1.022$\uparrow$} $\pm$ 0.03 \\\hline
Average & 0.131 $\pm$ 0.14
        & 0.311 $\pm$ 0.17
        & 0.403 $\pm$ 0.23
        & 0.435 $\pm$ 0.25
        & 0.450 $\pm$ 0.26
        & {\bf 0.486} $\pm$ 0.30 \\
$\bullet$ significance -- $p$ value                 &  &  0.000        &  0.000        &  0.000        &  0.000        & {\bf  0.000}        \\
$\bullet$ win/loss/draw  &  & 22/1/0  & 22/1/0  & 22/1/0  & 23/0/0  & {\bf 22/1/0} \\
$\bullet$ sig.win/sig.loss &  & 21/1 & 22/0 & 21/0 & 22/0 & {\bf 22/1} \\\hline
\end{tabular}
\caption{Relative average coverage per rule with standard
deviation ({\em COV $\pm$ sd}) for different variants of the
unordered algorithm using
  10-fold stratified cross-validation.}
\label{tab-results-Coverage-Unordered}
}
%\end{small}
\end{center}
%\vspace*{-1cm}
\end{table}
\setlength{\tabcolsep}{1.4pt}

%--------------------------------------------------------------------------%\begin{onecolumn}
%begin{table}[!ht]
\begin{table}[p]
\begin{center}
\vspace*{0.3cm}
{\footnotesize
%\begin{small}
\begin{tabular}{lrrrrrr}
\hline\noalign{\smallskip}
%\tiny
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD} \\
\#        & \multicolumn{1}{c}{\bf standard}
          & \multicolumn{1}{c}{\bf WRAcc}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
          & \multicolumn{1}{c}{\bf (add.)} \\
          & \multicolumn{1}{c}{\em SUP $\pm$ sd} &
            \multicolumn{1}{c}{\em SUP $\pm$ sd} &
            \multicolumn{1}{c}{\em SUP $\pm$ sd} &
            \multicolumn{1}{c}{\em SUP $\pm$ sd} &
            \multicolumn{1}{c}{\em SUP $\pm$ sd} &
            \multicolumn{1}{c}{\em SUP $\pm$ sd} \\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
  1 & 0.81 $\pm$ 0.09 & {\bf 0.89$\uparrow$} $\pm$ 0.02 & {\bf 0.87$\uparrow$} $\pm$ 0.00 & {\bf 0.97$\uparrow$} $\pm$ 0.01 & {\bf 0.84$\uparrow$} $\pm$ 0.00 & {\bf 0.89$\uparrow$} $\pm$ 0.04 \\
  2 & 0.88 $\pm$ 0.01 & 0.90                 $\pm$ 0.02 & 0.89                 $\pm$ 0.09 & 0.84                 $\pm$ 0.04 & {\bf 0.93$\uparrow$} $\pm$ 0.02 & 0.86                 $\pm$ 0.05 \\
  3 & 0.87 $\pm$ 0.05 & 0.87                 $\pm$ 0.09 & 0.84                 $\pm$ 0.05 & {\bf 0.93$\uparrow$} $\pm$ 0.02 & 0.84                 $\pm$ 0.07 & {\bf 0.95$\uparrow$} $\pm$ 0.01 \\
  4 & 0.87 $\pm$ 0.06 & 0.81$\downarrow$     $\pm$ 0.09 & 0.90                 $\pm$ 0.02 & 0.81$\downarrow$     $\pm$ 0.04 & {\bf 0.97$\uparrow$} $\pm$ 0.00 & {\bf 0.93$\uparrow$} $\pm$ 0.02 \\
  5 & 0.80 $\pm$ 0.01 & 0.82                 $\pm$ 0.03 & {\bf 0.92$\uparrow$} $\pm$ 0.06 & 0.85                 $\pm$ 0.01 & {\bf 0.95$\uparrow$} $\pm$ 0.01 & {\bf 0.87$\uparrow$} $\pm$ 0.05 \\
  6 & 0.90 $\pm$ 0.03 & 0.81$\downarrow$     $\pm$ 0.01 & {\bf 0.95$\uparrow$} $\pm$ 0.01 & 0.85                 $\pm$ 0.03 & {\bf 0.98$\uparrow$} $\pm$ 0.00 & 0.82$\downarrow$     $\pm$ 0.02 \\
  7 & 0.89 $\pm$ 0.03 & 0.88                 $\pm$ 0.03 & 0.90                 $\pm$ 0.02 & 0.81$\downarrow$     $\pm$ 0.07 & {\bf 0.97$\uparrow$} $\pm$ 0.01 & {\bf 0.96$\uparrow$} $\pm$ 0.01 \\
  8 & 0.84 $\pm$ 0.03 & 0.87                 $\pm$ 0.04 & {\bf 0.94$\uparrow$} $\pm$ 0.01 & 0.83                 $\pm$ 0.03 & 0.89                 $\pm$ 0.09 & {\bf 0.98$\uparrow$} $\pm$ 0.00 \\
  9 & 0.87 $\pm$ 0.10 & 0.81$\downarrow$     $\pm$ 0.02 & 0.85                 $\pm$ 0.10 & {\bf 0.94$\uparrow$} $\pm$ 0.00 & 0.90                 $\pm$ 0.02 & {\bf 0.99$\uparrow$} $\pm$ 0.00 \\
 10 & 0.84 $\pm$ 0.01 & 0.83                 $\pm$ 0.08 & 0.82                 $\pm$ 0.07 & {\bf 1.00$\uparrow$} $\pm$ 0.00 & {\bf 0.90$\uparrow$} $\pm$ 0.02 & {\bf 0.95$\uparrow$} $\pm$ 0.02 \\
 11 & 0.83 $\pm$ 0.03 & 0.85                 $\pm$ 0.07 & {\bf 0.96$\uparrow$} $\pm$ 0.01 & {\bf 0.95$\uparrow$} $\pm$ 0.01 & {\bf 0.89$\uparrow$} $\pm$ 0.09 & {\bf 0.98$\uparrow$} $\pm$ 0.01 \\
 12 & 0.82 $\pm$ 0.04 & {\bf 0.89$\uparrow$} $\pm$ 0.00 & 0.83                 $\pm$ 0.10 & {\bf 0.91$\uparrow$} $\pm$ 0.01 & {\bf 0.88$\uparrow$} $\pm$ 0.03 & {\bf 0.95$\uparrow$} $\pm$ 0.01 \\
 13 & 0.87 $\pm$ 0.10 & 0.90                 $\pm$ 0.06 & 0.81$\downarrow$     $\pm$ 0.02 & 0.80$\downarrow$     $\pm$ 0.09 & 0.85                 $\pm$ 0.04 & 0.85                 $\pm$ 0.03 \\
 14 & 0.84 $\pm$ 0.05 & 0.85                 $\pm$ 0.07 & 0.83                 $\pm$ 0.06 & {\bf 0.89$\uparrow$} $\pm$ 0.06 & {\bf 0.93$\uparrow$} $\pm$ 0.02 & 0.86                 $\pm$ 0.05 \\
 15 & 0.83 $\pm$ 0.04 & 0.80                 $\pm$ 0.07 & {\bf 0.96$\uparrow$} $\pm$ 0.01 & 0.86                 $\pm$ 0.09 & 0.80                 $\pm$ 0.08 & 0.81                 $\pm$ 0.00 \\
 16 & 0.85 $\pm$ 0.07 & 0.82                 $\pm$ 0.02 & {\bf 1.00$\uparrow$} $\pm$ 0.00 & 0.84                 $\pm$ 0.06 & {\bf 0.96$\uparrow$} $\pm$ 0.01 & 0.85                 $\pm$ 0.10 \\\hline
 17 & 0.86 $\pm$ 0.08 & {\bf 0.90$\uparrow$} $\pm$ 0.03 & 0.86                 $\pm$ 0.07 & 0.82                 $\pm$ 0.06 & {\bf 1.00$\uparrow$} $\pm$ 0.00 & 0.85                 $\pm$ 0.06 \\
 18 & 0.81 $\pm$ 0.06 & {\bf 0.85$\uparrow$} $\pm$ 0.07 & {\bf 0.96$\uparrow$} $\pm$ 0.01 & {\bf 0.89$\uparrow$} $\pm$ 0.05 & {\bf 0.95$\uparrow$} $\pm$ 0.01 & {\bf 0.97$\uparrow$} $\pm$ 0.00 \\
 19 & 0.83 $\pm$ 0.01 & 0.85                 $\pm$ 0.05 & {\bf 0.92$\uparrow$} $\pm$ 0.04 & {\bf 0.95$\uparrow$} $\pm$ 0.01 & {\bf 0.90$\uparrow$} $\pm$ 0.02 & 0.84                 $\pm$ 0.05 \\
 20 & 0.90 $\pm$ 0.06 & 0.82$\downarrow$     $\pm$ 0.07 & {\bf 0.99$\uparrow$} $\pm$ 0.00 & 0.90                 $\pm$ 0.03 & {\bf 0.99$\uparrow$} $\pm$ 0.00 & 0.90                 $\pm$ 0.04 \\
 21 & 0.81 $\pm$ 0.05 & 0.80                 $\pm$ 0.04 & {\bf 0.87$\uparrow$} $\pm$ 0.08 & {\bf 0.90$\uparrow$} $\pm$ 0.04 & {\bf 0.93$\uparrow$} $\pm$ 0.02 & 0.82                 $\pm$ 0.06 \\
 22 & 0.81 $\pm$ 0.02 & {\bf 0.89$\uparrow$} $\pm$ 0.06 & {\bf 0.94$\uparrow$} $\pm$ 0.02 & {\bf 0.96$\uparrow$} $\pm$ 0.01 & {\bf 1.00$\uparrow$} $\pm$ 0.00 & {\bf 0.96$\uparrow$} $\pm$ 0.01 \\
 23 & 0.82 $\pm$ 0.05 & 0.82                 $\pm$ 0.04 & {\bf 0.94$\uparrow$} $\pm$ 0.03 & {\bf 0.87$\uparrow$} $\pm$ 0.07 & {\bf 0.99$\uparrow$} $\pm$ 0.00 & {\bf 0.99$\uparrow$} $\pm$ 0.00 \\\hline
Average & 0.84 $\pm$ 0.03
        & 0.85 $\pm$ 0.03
        & 0.90 $\pm$ 0.06
        & 0.89 $\pm$ 0.06
        & {\bf 0.92} $\pm$ 0.06
        & 0.91 $\pm$ 0.06 \\
$\bullet$ significance -- $p$ value                 &  &   0.637       &  0.000        &  0.017        & {\bf  0.000}        &  0.001        \\
$\bullet$ win/loss/draw &  & 13/10/0  & 18/5/0  & 14/9/0  & {\bf 20/3/0} & 16/7/0  \\
$\bullet$ sig.win/sig.loss &  & 5/4 & 13/1 & 11/3 & {\bf 18/0} & 13/1 \\\hline
\end{tabular}
\caption{Overall rule set support with standard deviation ({\em SUP
    $\pm$ sd}) for different variants of the unordered algorithm using
  10-fold stratified cross-validation.}
\label{tab-results-Support-Unordered}
}
%\end{small}
\end{center}
%\vspace*{-1cm}
\end{table}
\setlength{\tabcolsep}{1.4pt}

%--------------------------------------------------------------------------
%\begin{onecolumn}
%begin{table}[!ht]
\begin{table}[p]
\begin{center}
\vspace*{0.3cm}
{\footnotesize
%\begin{small}
\begin{tabular}{lrrrrrr}
\hline\noalign{\smallskip}
%\tiny
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD} \\
\#        & \multicolumn{1}{c}{\bf standard}
          & \multicolumn{1}{c}{\bf WRAcc}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
          & \multicolumn{1}{c}{\bf (add.)} \\
          & \multicolumn{1}{c}{\em SIZE $\pm$ sd} &
            \multicolumn{1}{c}{\em SIZE $\pm$ sd} &
            \multicolumn{1}{c}{\em SIZE $\pm$ sd} &
            \multicolumn{1}{c}{\em SIZE $\pm$ sd} &
            \multicolumn{1}{c}{\em SIZE $\pm$ sd} &
            \multicolumn{1}{c}{\em SIZE $\pm$ sd} \\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
  1 & 12.4 $\pm$ 1.95 & {\bf 2.0$\uparrow$}  $\pm$ 0.75 & {\bf 2.7$\uparrow$}  $\pm$ 0.02 & {\bf 2.6$\uparrow$}  $\pm$ 0.87 & {\bf 2.2$\uparrow$}  $\pm$ 0.85 & {\bf 3.5$\uparrow$}  $\pm$ 0.79 \\
  2 & 12.6 $\pm$ 1.04 & {\bf 8.8$\uparrow$}  $\pm$ 0.95 & {\bf 7.9$\uparrow$}  $\pm$ 0.50 & {\bf 8.5$\uparrow$}  $\pm$ 1.75 & {\bf 9.0$\uparrow$}  $\pm$ 0.24 & {\bf 9.2$\uparrow$}  $\pm$ 1.24 \\
  3 & 1.8  $\pm$ 0.10 & 2.0                  $\pm$ 0.41 & 2.0                  $\pm$ 0.70 & 2.7 $\downarrow$     $\pm$ 0.44 & 1.9                  $\pm$ 0.27 & 1.8                  $\pm$ 0.29 \\
  4 & 14.6 $\pm$ 1.81 & {\bf 7.9$\uparrow$}  $\pm$ 1.78 & {\bf 8.1$\uparrow$}  $\pm$ 1.02 & {\bf 7.9$\uparrow$}  $\pm$ 0.97 & {\bf 8.5$\uparrow$}  $\pm$ 0.47 & {\bf 8.5$\uparrow$}  $\pm$ 0.41 \\
  5 & 12.8 $\pm$ 1.56 & {\bf 5.2$\uparrow$}  $\pm$ 0.79 & {\bf 6.0$\uparrow$}  $\pm$ 0.68 & {\bf 5.6$\uparrow$}  $\pm$ 1.35 & {\bf 5.4$\uparrow$}  $\pm$ 0.30 & {\bf 4.6$\uparrow$}  $\pm$ 0.86 \\
  6 & 3.7  $\pm$ 1.37 & {\bf 2.5$\uparrow$}  $\pm$ 0.79 & 3.1                  $\pm$ 0.72 & 3.8                  $\pm$ 1.61 & 4.7 $\downarrow$     $\pm$ 1.22 & 3.4                  $\pm$ 0.02 \\
  7 & 15.1 $\pm$ 1.89 & {\bf 7.8$\uparrow$}  $\pm$ 1.49 & {\bf 8.4$\uparrow$}  $\pm$ 1.32 & {\bf 8.7$\uparrow$}  $\pm$ 0.46 & {\bf 9.1$\uparrow$}  $\pm$ 1.26 & {\bf 8.8$\uparrow$}  $\pm$ 1.13 \\
  8 & 6.4  $\pm$ 1.53 & {\bf 3.0$\uparrow$}  $\pm$ 1.20 & {\bf 2.9$\uparrow$}  $\pm$ 0.98 & {\bf 2.7$\uparrow$}  $\pm$ 0.67 & {\bf 2.7$\uparrow$}  $\pm$ 0.90 & {\bf 1.8$\uparrow$}  $\pm$ 0.38 \\
  9 & 3.0  $\pm$ 0.29 & {\bf 2.1$\uparrow$}  $\pm$ 0.50 & {\bf 1.7$\uparrow$}  $\pm$ 0.93 & 2.7                  $\pm$ 0.53 & 3.6 $\downarrow$     $\pm$ 1.83 & 2.7                  $\pm$ 0.00 \\
 10 & 10.1 $\pm$ 1.02 & {\bf 3.9$\uparrow$}  $\pm$ 0.31 & {\bf 3.9$\uparrow$}  $\pm$ 0.85 & {\bf 3.4$\uparrow$}  $\pm$ 1.10 & {\bf 3.3$\uparrow$}  $\pm$ 1.90 & {\bf 2.5$\uparrow$}  $\pm$ 0.54 \\
 11 & 7.6  $\pm$ 1.01 & {\bf 3.0$\uparrow$}  $\pm$ 1.78 & {\bf 3.9$\uparrow$}  $\pm$ 1.84 & {\bf 4.0$\uparrow$}  $\pm$ 0.18 & {\bf 3.6$\uparrow$}  $\pm$ 0.87 & {\bf 4.2$\uparrow$}  $\pm$ 0.41 \\
 12 & 3.8  $\pm$ 1.24 & {\bf 3.0$\uparrow$}  $\pm$ 1.24 & {\bf 3.2$\uparrow$}  $\pm$ 0.42 & {\bf 3.4$\uparrow$}  $\pm$ 0.39 & {\bf 2.9$\uparrow$}  $\pm$ 0.05 & 3.6                  $\pm$ 0.69 \\
 13 & 4.7  $\pm$ 1.30 & {\bf 3.1$\uparrow$}  $\pm$ 1.15 & {\bf 3.4$\uparrow$}  $\pm$ 0.54 & {\bf 3.9$\uparrow$}  $\pm$ 0.98 & 4.6                  $\pm$ 1.19 & 4.5                  $\pm$ 0.71 \\
 14 & 5.2  $\pm$ 0.90 & {\bf 2.7$\uparrow$}  $\pm$ 0.91 & {\bf 2.1$\uparrow$}  $\pm$ 0.95 & {\bf 1.9$\uparrow$}  $\pm$ 0.10 & {\bf 1.7$\uparrow$}  $\pm$ 1.73 & {\bf 2.1$\uparrow$}  $\pm$ 0.78 \\
 15 & 21.2 $\pm$ 3.48 & {\bf 10.5$\uparrow$} $\pm$ 1.85 & {\bf 11.2$\uparrow$} $\pm$ 1.12 & {\bf 10.3$\uparrow$} $\pm$ 1.99 & {\bf 9.6$\uparrow$}  $\pm$ 1.32 & {\bf 10.2$\uparrow$} $\pm$ 1.30 \\
 16 & 7.1  $\pm$ 1.59 & {\bf 2.0$\uparrow$}  $\pm$ 0.81 & {\bf 2.4$\uparrow$}  $\pm$ 0.56 & {\bf 2.4$\uparrow$}  $\pm$ 0.75 & {\bf 2.9$\uparrow$}  $\pm$ 0.56 & {\bf 1.8$\uparrow$}  $\pm$ 0.45 \\\hline
 17 & 28.7 $\pm$ 3.89 & {\bf 9.9$\uparrow$}  $\pm$ 1.22 & {\bf 9.4$\uparrow$}  $\pm$ 1.61 & {\bf 8.9$\uparrow$}  $\pm$ 1.80 & {\bf 9.5$\uparrow$}  $\pm$ 1.03 & {\bf 8.3$\uparrow$}  $\pm$ 1.17 \\
 18 & 83.8 $\pm$ 5.37 & {\bf 10.9$\uparrow$} $\pm$ 2.37 & {\bf 11.3$\uparrow$} $\pm$ 2.78 & {\bf 11.8$\uparrow$} $\pm$ 1.45 & {\bf 11.7$\uparrow$} $\pm$ 1.67 & {\bf 12.8$\uparrow$} $\pm$ 1.74 \\
 19 & 12.9 $\pm$ 1.68 & {\bf 7.7$\uparrow$}  $\pm$ 1.00 & {\bf 8.6$\uparrow$}  $\pm$ 1.21 & {\bf 9.1$\uparrow$}  $\pm$ 1.85 & {\bf 8.4$\uparrow$}  $\pm$ 1.09 & {\bf 10.1$\uparrow$} $\pm$ 1.83 \\
 20 & 32.8 $\pm$ 2.64 & {\bf 8.7$\uparrow$}  $\pm$ 1.82 & {\bf 8.9$\uparrow$}  $\pm$ 1.48 & {\bf 9.8$\uparrow$}  $\pm$ 1.01 & {\bf 10.5$\uparrow$} $\pm$ 1.37 & {\bf 9.2$\uparrow$}  $\pm$ 1.49 \\
 21 & 35.1 $\pm$ 3.54 & {\bf 19.6$\uparrow$} $\pm$ 1.80 & {\bf 19.3$\uparrow$} $\pm$ 2.91 & {\bf 19.7$\uparrow$} $\pm$ 2.99 & {\bf 19.8$\uparrow$} $\pm$ 2.58 & {\bf 19.2$\uparrow$} $\pm$ 2.90 \\
 22 & 77.3 $\pm$ 4.07 & {\bf 12.2$\uparrow$} $\pm$ 1.79 & {\bf 11.4$\uparrow$} $\pm$ 2.87 & {\bf 12.4$\uparrow$} $\pm$ 2.29 & {\bf 12.4$\uparrow$} $\pm$ 2.09 & {\bf 11.7$\uparrow$} $\pm$ 2.81 \\
 23 & 5.5  $\pm$ 1.26 & {\bf 3.0$\uparrow$}  $\pm$ 0.36 & {\bf 2.1$\uparrow$}  $\pm$ 0.70 & {\bf 2.1$\uparrow$}  $\pm$ 0.57 & {\bf 1.2$\uparrow$}  $\pm$ 0.73 & {\bf 1.4$\uparrow$}  $\pm$ 0.90 \\\hline
Average & 18.18 $\pm$ 21.77
        & {\bf  6.15} $\pm$ 4.49
        & 6.25 $\pm$ 4.42
        & 6.45 $\pm$ 4.48
        & 6.49 $\pm$ 4.57
        & 6.35 $\pm$ 4.58 \\
$\bullet$ significance -- $p$ value                 &  & {\bf  0.006}        &  0.007        &  0.007        &  0.007        &  0.007        \\
$\bullet$ win/loss/draw  &  & {\bf 22/1/0} & 22/1/0  & 21/2/0  & 20/3/0  & 23/0/0  \\
$\bullet$ sig.win/sig.loss &  & {\bf 22/0} & 21/0 & 20/1 & 19/2 & 18/0 \\\hline
\end{tabular}
\caption{Average rule set sizes with standard deviation ({\em SIZE
    $\pm$ sd}) for different variants of the unordered algorithm using
  10-fold stratified cross-validation.}
\label{tab-results-Size-Unordered}
}
%\end{small}
\end{center}
%\vspace*{-1cm}
\end{table}
\setlength{\tabcolsep}{1.4pt}

%-------------------------------------------------------------------------
%\begin{onecolumn}
%begin{table}[!ht]
\begin{table}[p]
\begin{center}
\vspace*{0.3cm}
{\footnotesize
%\begin{small}
\begin{tabular}{lrrrrrr}
\hline\noalign{\smallskip}
%\tiny
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD} \\
\#        & \multicolumn{1}{c}{\bf standard}
          & \multicolumn{1}{c}{\bf WRAcc}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
          & \multicolumn{1}{c}{\bf (add.)} \\
          & \multicolumn{1}{c}{\em SIG $\pm$ sd} &
            \multicolumn{1}{c}{\em SIG $\pm$ sd} &
            \multicolumn{1}{c}{\em SIG $\pm$ sd} &
            \multicolumn{1}{c}{\em SIG $\pm$ sd} &
            \multicolumn{1}{c}{\em SIG $\pm$ sd} &
            \multicolumn{1}{c}{\em SIG $\pm$ sd} \\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
  1 & 2.0 $\pm$ 0.05 & {\bf 7.8$\uparrow$}  $\pm$ 1.49 & {\bf 14.6$\uparrow$} $\pm$ 1.05 & {\bf 24.0$\uparrow$} $\pm$ 1.01 & {\bf 15.6$\uparrow$} $\pm$ 1.54 & {\bf 4.6$\uparrow$}  $\pm$ 0.52 \\
  2 & 2.7 $\pm$ 0.10 & {\bf 13.3$\uparrow$} $\pm$ 1.69 & {\bf 27.1$\uparrow$} $\pm$ 3.37 & 2.1                  $\pm$ 0.02 & {\bf 20.5$\uparrow$} $\pm$ 2.45 & {\bf 26.6$\uparrow$} $\pm$ 3.43 \\
  3 & 2.1 $\pm$ 0.01 & {\bf 7.8$\uparrow$}  $\pm$ 0.64 & {\bf 13.3$\uparrow$} $\pm$ 1.39 & 2.5                  $\pm$ 0.01 & {\bf 21.2$\uparrow$} $\pm$ 2.55 & {\bf 22.9$\uparrow$} $\pm$ 2.43 \\
  4 & 2.4 $\pm$ 0.06 & {\bf 9.1$\uparrow$}  $\pm$ 0.58 & {\bf 14.1$\uparrow$} $\pm$ 1.72 & {\bf 16.9$\uparrow$} $\pm$ 1.28 & {\bf 22.5$\uparrow$} $\pm$ 2.49 & {\bf 30.2$\uparrow$} $\pm$ 3.98 \\
  5 & 2.0 $\pm$ 0.01 & {\bf 15.8$\uparrow$} $\pm$ 1.07 & {\bf 14.9$\uparrow$} $\pm$ 1.95 & {\bf 11.0$\uparrow$} $\pm$ 1.43 & {\bf 15.2$\uparrow$} $\pm$ 1.85 & 2.1                  $\pm$ 0.01 \\
  6 & 1.9 $\pm$ 0.03 & {\bf 10.0$\uparrow$} $\pm$ 1.63 & {\bf 11.0$\uparrow$} $\pm$ 1.12 & {\bf 30.5$\uparrow$} $\pm$ 2.12 & {\bf 30.1$\uparrow$} $\pm$ 2.27 & {\bf 23.1$\uparrow$} $\pm$ 2.97 \\
  7 & 2.0 $\pm$ 0.02 & 2.7                  $\pm$ 0.83 & {\bf 19.8$\uparrow$} $\pm$ 1.21 & {\bf 17.7$\uparrow$} $\pm$ 1.63 & {\bf 11.1$\uparrow$} $\pm$ 1.03 & {\bf 16.3$\uparrow$} $\pm$ 1.49 \\
  8 & 1.9 $\pm$ 0.09 & {\bf 4.6$\uparrow$}  $\pm$ 0.59 & {\bf 23.2$\uparrow$} $\pm$ 1.82 & {\bf 5.3$\uparrow$}  $\pm$ 0.36 & {\bf 4.0$\uparrow$}  $\pm$ 0.03 & {\bf 30.6$\uparrow$} $\pm$ 2.96 \\
  9 & 2.7 $\pm$ 0.03 & {\bf 9.7$\uparrow$}  $\pm$ 0.86 & {\bf 12.3$\uparrow$} $\pm$ 1.00 & {\bf 9.3$\uparrow$}  $\pm$ 0.65 & {\bf 8.5$\uparrow$}  $\pm$ 0.89 & {\bf 25.0$\uparrow$} $\pm$ 2.60 \\
 10 & 1.4 $\pm$ 0.04 & {\bf 3.6$\uparrow$}  $\pm$ 0.74 & {\bf 5.8$\uparrow$}  $\pm$ 0.48 & {\bf 28.3$\uparrow$} $\pm$ 2.27 & {\bf 24.9$\uparrow$} $\pm$ 2.27 & {\bf 13.5$\uparrow$} $\pm$ 1.84 \\
 11 & 2.0 $\pm$ 0.04 & 1.8                  $\pm$ 0.07 & {\bf 16.7$\uparrow$} $\pm$ 1.42 & {\bf 23.9$\uparrow$} $\pm$ 2.41 & {\bf 30.9$\uparrow$} $\pm$ 2.18 & {\bf 14.9$\uparrow$} $\pm$ 1.52 \\
 12 & 1.9 $\pm$ 0.03 & {\bf 7.1$\uparrow$}  $\pm$ 0.07 & {\bf 17.0$\uparrow$} $\pm$ 1.61 & 1.3                  $\pm$ 0.09 & {\bf 17.6$\uparrow$} $\pm$ 1.45 & {\bf 4.0$\uparrow$}  $\pm$ 0.00 \\
 13 & 2.1 $\pm$ 0.00 & {\bf 15.1$\uparrow$} $\pm$ 1.80 & {\bf 19.4$\uparrow$} $\pm$ 1.77 & {\bf 21.9$\uparrow$} $\pm$ 2.38 & {\bf 21.4$\uparrow$} $\pm$ 2.39 & {\bf 9.7$\uparrow$}  $\pm$ 0.61 \\
 14 & 2.5 $\pm$ 0.08 & {\bf 14.9$\uparrow$} $\pm$ 1.93 & {\bf 18.0$\uparrow$} $\pm$ 1.57 & {\bf 13.9$\uparrow$} $\pm$ 1.28 & 3.0                  $\pm$ 0.09 & {\bf 18.1$\uparrow$} $\pm$ 1.73 \\
 15 & 2.5 $\pm$ 0.05 & {\bf 4.2$\uparrow$}  $\pm$ 0.42 & {\bf 17.5$\uparrow$} $\pm$ 1.79 & {\bf 5.7$\uparrow$}  $\pm$ 0.46 & {\bf 21.9$\uparrow$} $\pm$ 2.83 & {\bf 26.5$\uparrow$} $\pm$ 2.22 \\
 16 & 2.6 $\pm$ 0.04 & {\bf 11.7$\uparrow$} $\pm$ 1.90 & {\bf 9.6$\uparrow$}  $\pm$ 0.56 & {\bf 22.7$\uparrow$} $\pm$ 2.59 & 2.3                  $\pm$ 0.08 & {\bf 6.0$\uparrow$}  $\pm$ 0.00 \\\hline
 17 & 2.7 $\pm$ 0.03 & {\bf 4.8$\uparrow$}  $\pm$ 0.53 & {\bf 11.7$\uparrow$} $\pm$ 1.67 & {\bf 21.8$\uparrow$} $\pm$ 2.55 & {\bf 15.0$\uparrow$} $\pm$ 1.82 & {\bf 24.3$\uparrow$} $\pm$ 2.26 \\
 18 & 1.5 $\pm$ 0.00 & {\bf 14.1$\uparrow$} $\pm$ 1.11 & {\bf 6.0$\uparrow$}  $\pm$ 0.93 & {\bf 26.8$\uparrow$} $\pm$ 2.53 & {\bf 12.6$\uparrow$} $\pm$ 1.35 & {\bf 19.3$\uparrow$} $\pm$ 1.09 \\
 19 & 1.0 $\pm$ 0.07 & {\bf 2.4$\uparrow$}  $\pm$ 0.01 & {\bf 22.0$\uparrow$} $\pm$ 1.20 & {\bf 17.0$\uparrow$} $\pm$ 1.78 & {\bf 16.4$\uparrow$} $\pm$ 1.74 & {\bf 9.1$\uparrow$}  $\pm$ 0.02 \\
 20 & 1.5 $\pm$ 0.00 & {\bf 16.0$\uparrow$} $\pm$ 2.52 & {\bf 24.3$\uparrow$} $\pm$ 1.52 & {\bf 11.4$\uparrow$} $\pm$ 1.25 & {\bf 29.9$\uparrow$} $\pm$ 3.25 & {\bf 21.7$\uparrow$} $\pm$ 2.88 \\
 21 & 2.4 $\pm$ 0.02 & {\bf 6.8$\uparrow$}  $\pm$ 0.88 & {\bf 15.6$\uparrow$} $\pm$ 1.98 & {\bf 12.9$\uparrow$} $\pm$ 1.47 & {\bf 8.2$\uparrow$}  $\pm$ 0.06 & {\bf 30.6$\uparrow$} $\pm$ 2.39 \\
 22 & 2.6 $\pm$ 0.04 & {\bf 9.7$\uparrow$}  $\pm$ 1.56 & {\bf 3.4$\uparrow$}  $\pm$ 0.09 & {\bf 14.2$\uparrow$} $\pm$ 1.20 & {\bf 7.1$\uparrow$}  $\pm$ 0.47 & {\bf 20.2$\uparrow$} $\pm$ 2.71 \\
 23 & 2.0 $\pm$ 0.07 & {\bf 13.5$\uparrow$} $\pm$ 1.57 & {\bf 20.7$\uparrow$} $\pm$ 1.93 & {\bf 2.7$\uparrow$}  $\pm$ 0.02 & {\bf 29.4$\uparrow$} $\pm$ 3.51 & {\bf 25.7$\uparrow$} $\pm$ 2.48 \\\hline
Average &  2.11 $\pm$ 0.46
        &  8.97 $\pm$ 4.66
        & 15.57 $\pm$ 6.05
        & 14.95 $\pm$ 9.02
        & 16.92 $\pm$ 8.90
        & {\bf 18.47} $\pm$ 9.00 \\
$\bullet$ significance -- $p$ value                 &  &  0.000        &  0.000        &  0.000        &  0.000        & {\bf  0.000}        \\
$\bullet$ win/loss/draw  &  & 22/1/0 & 23/0/0  & 21/2/0  & 22/1/0  & {\bf 23/0/0} \\
$\bullet$ sig.win/sig.loss &  & 21/0 & 23/0 & 20/0 & 21/0 & {\bf 22/0} \\\hline
\end{tabular}
\caption{Average rule significance with standard deviation ({\em
  SIG $\pm$ sd}) for different variants of the unordered algorithm using
  10-fold stratified cross-validation.}
\label{tab-results-Significance-Unordered}
}
%\end{small}
\end{center}
%\vspace*{-1cm}
\end{table}
\setlength{\tabcolsep}{1.4pt}


%-------------------------------------------------------------------------
%\begin{onecolumn}
%begin{table}[!ht]
\begin{table}[p]
\begin{center}
\vspace*{0.3cm}
{\footnotesize
%\begin{small}
\begin{tabular}{lrrrrrr}
\hline\noalign{\smallskip}
%\tiny
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD} \\
\#        & \multicolumn{1}{c}{\bf standard}
          & \multicolumn{1}{c}{\bf WRAcc}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
          & \multicolumn{1}{c}{\bf (add.)} \\
          & \multicolumn{1}{c}{\em WRACC $\pm$ sd} &
            \multicolumn{1}{c}{\em WRACC $\pm$ sd} &
            \multicolumn{1}{c}{\em WRACC $\pm$ sd} &
            \multicolumn{1}{c}{\em WRACC $\pm$ sd} &
            \multicolumn{1}{c}{\em WRACC $\pm$ sd} &
            \multicolumn{1}{c}{\em WRACC $\pm$ sd} \\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
  1 &  0.022 $\pm$ 0.09 &  {\bf 0.148$\uparrow$} $\pm$ 0.03 &  {\bf 0.186$\uparrow$} $\pm$ 0.09 &  {\bf 0.185$\uparrow$} $\pm$ 0.04 &  {\bf 0.181$\uparrow$} $\pm$ 0.07 &  {\bf 0.162$\uparrow$} $\pm$ 0.01 \\
  2 &  0.034 $\pm$ 0.04 &  {\bf 0.063$\uparrow$} $\pm$ 0.04 &  {\bf 0.095$\uparrow$} $\pm$ 0.02 &  {\bf 0.079$\uparrow$} $\pm$ 0.01 &  {\bf 0.093$\uparrow$} $\pm$ 0.07 &  {\bf 0.111$\uparrow$} $\pm$ 0.04 \\
  3 & -0.016 $\pm$ 0.08 & -0.012                 $\pm$ 0.01 & -0.005                 $\pm$ 0.03 & -0.006                 $\pm$ 0.09 & -0.001                 $\pm$ 0.02 & -0.012                 $\pm$ 0.01 \\
  4 &  0.020 $\pm$ 0.04 &  {\bf 0.210$\uparrow$} $\pm$ 0.02 &  {\bf 0.228$\uparrow$} $\pm$ 0.02 &  {\bf 0.233$\uparrow$} $\pm$ 0.04 &  {\bf 0.224$\uparrow$} $\pm$ 0.03 &  {\bf 0.224$\uparrow$} $\pm$ 0.10 \\
  5 &  0.013 $\pm$ 0.06 &  {\bf 0.065$\uparrow$} $\pm$ 0.06 &  {\bf 0.085$\uparrow$} $\pm$ 0.07 &  {\bf 0.099$\uparrow$} $\pm$ 0.04 &  {\bf 0.086$\uparrow$} $\pm$ 0.07 &  {\bf 0.092$\uparrow$} $\pm$ 0.03 \\
  6 &  0.058 $\pm$ 0.07 &  {\bf 0.099$\uparrow$} $\pm$ 0.10 &  {\bf 0.174$\uparrow$} $\pm$ 0.05 &  {\bf 0.208$\uparrow$} $\pm$ 0.00 &  {\bf 0.213$\uparrow$} $\pm$ 0.01 &  {\bf 0.243$\uparrow$} $\pm$ 0.10 \\
  7 &  0.012 $\pm$ 0.02 &  {\bf 0.020$\uparrow$} $\pm$ 0.01 &  {\bf 0.034$\uparrow$} $\pm$ 0.00 &  {\bf 0.040$\uparrow$} $\pm$ 0.05 &  {\bf 0.034$\uparrow$} $\pm$ 0.08 &  {\bf 0.034$\uparrow$} $\pm$ 0.08 \\
  8 &  0.026 $\pm$ 0.04 &  {\bf 0.065$\uparrow$} $\pm$ 0.04 &  {\bf 0.124$\uparrow$} $\pm$ 0.02 &  {\bf 0.104$\uparrow$} $\pm$ 0.06 &  {\bf 0.104$\uparrow$} $\pm$ 0.09 &  {\bf 0.122$\uparrow$} $\pm$ 0.03 \\
  9 &  0.004 $\pm$ 0.07 &  {\bf 0.018$\uparrow$} $\pm$ 0.04 &  {\bf 0.057$\uparrow$} $\pm$ 0.10 &  {\bf 0.073$\uparrow$} $\pm$ 0.09 &  {\bf 0.066$\uparrow$} $\pm$ 0.04 &  {\bf 0.049$\uparrow$} $\pm$ 0.02 \\
 10 &  0.013 $\pm$ 0.04 &  {\bf 0.067$\uparrow$} $\pm$ 0.02 &  {\bf 0.076$\uparrow$} $\pm$ 0.01 &  {\bf 0.073$\uparrow$} $\pm$ 0.09 &  {\bf 0.076$\uparrow$} $\pm$ 0.04 &  {\bf 0.072$\uparrow$} $\pm$ 0.07 \\
 11 &  0.041 $\pm$ 0.02 &  {\bf 0.065$\uparrow$} $\pm$ 0.03 &  {\bf 0.099$\uparrow$} $\pm$ 0.04 &  {\bf 0.095$\uparrow$} $\pm$ 0.05 &  {\bf 0.104$\uparrow$} $\pm$ 0.10 &  {\bf 0.145$\uparrow$} $\pm$ 0.00 \\
 12 &  0.024 $\pm$ 0.04 &  0.024                 $\pm$ 0.05 &  {\bf 0.062$\uparrow$} $\pm$ 0.02 &  {\bf 0.042$\uparrow$} $\pm$ 0.02 &  {\bf 0.052$\uparrow$} $\pm$ 0.03 &  {\bf 0.045$\uparrow$} $\pm$ 0.06 \\
 13 &  0.024 $\pm$ 0.03 &  {\bf 0.056$\uparrow$} $\pm$ 0.03 &  {\bf 0.114$\uparrow$} $\pm$ 0.10 &  {\bf 0.085$\uparrow$} $\pm$ 0.04 &  {\bf 0.065$\uparrow$} $\pm$ 0.07 &  {\bf 0.092$\uparrow$} $\pm$ 0.03 \\
 14 &  0.009 $\pm$ 0.10 &  {\bf 0.038$\uparrow$} $\pm$ 0.10 &  {\bf 0.053$\uparrow$} $\pm$ 0.03 &  {\bf 0.082$\uparrow$} $\pm$ 0.10 &  {\bf 0.082$\uparrow$} $\pm$ 0.02 &  {\bf 0.085$\uparrow$} $\pm$ 0.08 \\
 15 &  0.015 $\pm$ 0.07 &  {\bf 0.030$\uparrow$} $\pm$ 0.07 &  {\bf 0.036$\uparrow$} $\pm$ 0.09 &  {\bf 0.041$\uparrow$} $\pm$ 0.03 &  {\bf 0.055$\uparrow$} $\pm$ 0.08 &  {\bf 0.032$\uparrow$} $\pm$ 0.06 \\
 16 &  0.017 $\pm$ 0.00 &  {\bf 0.095$\uparrow$} $\pm$ 0.10 &  {\bf 0.117$\uparrow$} $\pm$ 0.04 &  {\bf 0.129$\uparrow$} $\pm$ 0.04 &  {\bf 0.127$\uparrow$} $\pm$ 0.06 &  {\bf 0.138$\uparrow$} $\pm$ 0.02 \\\hline
 17 &  0.005 $\pm$ 0.03 &  {\bf 0.048$\uparrow$} $\pm$ 0.07 &  {\bf 0.051$\uparrow$} $\pm$ 0.02 &  {\bf 0.073$\uparrow$} $\pm$ 0.08 &  {\bf 0.083$\uparrow$} $\pm$ 0.02 &  {\bf 0.073$\uparrow$} $\pm$ 0.09 \\
 18 &  0.009 $\pm$ 0.06 &  {\bf 0.030$\uparrow$} $\pm$ 0.00 &  {\bf 0.037$\uparrow$} $\pm$ 0.01 &  {\bf 0.032$\uparrow$} $\pm$ 0.00 &  {\bf 0.034$\uparrow$} $\pm$ 0.07 &  {\bf 0.045$\uparrow$} $\pm$ 0.03 \\
 19 &  0.007 $\pm$ 0.07 &  {\bf 0.060$\uparrow$} $\pm$ 0.00 &  {\bf 0.081$\uparrow$} $\pm$ 0.08 &  {\bf 0.133$\uparrow$} $\pm$ 0.05 &  {\bf 0.132$\uparrow$} $\pm$ 0.03 &  {\bf 0.147$\uparrow$} $\pm$ 0.04 \\
 20 &  0.004 $\pm$ 0.01 & -0.045$\downarrow$     $\pm$ 0.10 & -0.042$\downarrow$     $\pm$ 0.04 & -0.048$\downarrow$     $\pm$ 0.02 & -0.042$\downarrow$     $\pm$ 0.03 & -0.051$\downarrow$     $\pm$ 0.06 \\
 21 &  0.015 $\pm$ 0.08 &  0.015                 $\pm$ 0.03 &  {\bf 0.024$\uparrow$} $\pm$ 0.04 &  {\bf 0.039$\uparrow$} $\pm$ 0.08 &  {\bf 0.042$\uparrow$} $\pm$ 0.06 &  {\bf 0.045$\uparrow$} $\pm$ 0.05 \\
 22 &  0.001 $\pm$ 0.03 &  {\bf 0.045$\uparrow$} $\pm$ 0.06 &  {\bf 0.054$\uparrow$} $\pm$ 0.05 &  {\bf 0.054$\uparrow$} $\pm$ 0.09 &  {\bf 0.054$\uparrow$} $\pm$ 0.05 &  {\bf 0.049$\uparrow$} $\pm$ 0.05 \\
 23 &  0.033 $\pm$ 0.01 &  {\bf 0.076$\uparrow$} $\pm$ 0.05 &  {\bf 0.089$\uparrow$} $\pm$ 0.03 &  {\bf 0.144$\uparrow$} $\pm$ 0.05 &  {\bf 0.149$\uparrow$} $\pm$ 0.06 &  {\bf 0.167$\uparrow$} $\pm$ 0.01 \\\hline
Average & 0.017 $\pm$ 0.02
        & 0.056 $\pm$ 0.05
        & 0.079 $\pm$ 0.06
        & 0.086 $\pm$ 0.07
        & 0.088 $\pm$ 0.06
        & {\bf 0.092} $\pm$ 0.07 \\
$\bullet$ significance -- $p$ value                 &  &  0.001        &  0.000        &  0.000        &  0.000        & {\bf  0.000}        \\
$\bullet$ win/loss/draw  &  & 20/1/2  & 22/1/0  & 22/1/0  & 22/1/0  & {\bf 22/1/0} \\
$\bullet$ sig.win/sig.loss &  & 19/1 & 21/1 & 21/1 & 21/1 & {\bf 21/1} \\\hline
\end{tabular}
\caption{Average rule unusualness with standard deviation ({\em
  WRACC $\pm$ sd}) for different variants of the unordered algorithm using
  10-fold stratified cross-validation.}
\label{tab-results-Unusualness-Unordered}
}
%\end{small}
\end{center}
%\vspace*{-1cm}
\end{table}
\setlength{\tabcolsep}{1.4pt}


%--------------------------------------------------------------------------
%\begin{onecolumn}
%begin{table}[!ht]
\begin{table}[p]
\begin{center}
\vspace*{0.3cm}
{\footnotesize
%\begin{small}
\begin{tabular}{lrrrrrr}
\hline\noalign{\smallskip}
%\tiny
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD} \\
\#        & \multicolumn{1}{c}{\bf standard}
          & \multicolumn{1}{c}{\bf WRAcc}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
          & \multicolumn{1}{c}{\bf (add.)} \\
          & \multicolumn{1}{c}{\em ACC $\pm$ sd} &
            \multicolumn{1}{c}{\em ACC $\pm$ sd} &
            \multicolumn{1}{c}{\em ACC $\pm$ sd} &
            \multicolumn{1}{c}{\em ACC $\pm$ sd} &
            \multicolumn{1}{c}{\em ACC $\pm$ sd} &
            \multicolumn{1}{c}{\em ACC $\pm$ sd} \\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
  1 & 81.62 $\pm$ 3.55 & {\bf 85.53$\uparrow$} $\pm$ 0.14 & {\bf 89.27$\uparrow$} $\pm$ 8.04 & {\bf 87.61$\uparrow$} $\pm$ 8.71 & {\bf 87.81$\uparrow$} $\pm$ 6.54 & {\bf 88.35$\uparrow$} $\pm$ 8.60 \\
  2 & 92.28 $\pm$ 1.07 & 92.13                 $\pm$ 5.95 & 95.80                 $\pm$ 1.21 & 95.56                 $\pm$ 3.15 & 92.53                 $\pm$ 1.52 & 92.60                 $\pm$ 2.28 \\
  3 & 82.45 $\pm$ 3.89 & 81.36                 $\pm$ 1.30 & 84.13                 $\pm$ 8.88 & 84.07                 $\pm$ 6.11 & 84.81                 $\pm$ 1.06 & 81.46                 $\pm$ 2.24 \\
  4 & 94.18 $\pm$ 3.71 & 94.34                 $\pm$ 2.25 & 97.19                 $\pm$ 0.35 & 97.37                 $\pm$ 0.42 & 96.54                 $\pm$ 1.77 & 96.08                 $\pm$ 1.22 \\
  5 & 72.77 $\pm$ 9.33 & 73.81                 $\pm$ 0.91 & {\bf 78.66$\uparrow$} $\pm$ 8.65 & {\bf 78.80$\uparrow$} $\pm$ 0.04 & {\bf 78.81$\uparrow$} $\pm$ 5.55 & 74.12                 $\pm$ 9.97 \\
  6 & 68.71 $\pm$ 1.79 & 67.12                 $\pm$ 6.55 & 68.62                 $\pm$ 0.96 & 70.08                 $\pm$ 6.28 & {\bf 71.20$\uparrow$} $\pm$ 9.94 & 68.75                 $\pm$ 5.32 \\
  7 & 72.40 $\pm$ 7.60 & 71.40                 $\pm$ 7.57 & 73.73                 $\pm$ 0.72 & {\bf 75.82$\uparrow$} $\pm$ 8.07 & 74.67                 $\pm$ 5.85 & 72.40                 $\pm$ 7.36 \\
  8 & 74.10 $\pm$ 4.15 & 77.06                 $\pm$ 7.06 & {\bf 79.64$\uparrow$} $\pm$ 6.98 & 77.53                 $\pm$ 4.77 & {\bf 78.48$\uparrow$} $\pm$ 3.16 & {\bf 78.03$\uparrow$} $\pm$ 2.70 \\
  9 & 80.74 $\pm$ 7.59 & 83.26                 $\pm$ 0.83 & {\bf 87.87$\uparrow$} $\pm$ 2.29 & {\bf 87.75$\uparrow$} $\pm$ 0.36 & {\bf 86.97$\uparrow$} $\pm$ 6.88 & {\bf 86.14$\uparrow$} $\pm$ 1.99 \\
 10 & 98.58 $\pm$ 0.60 & 98.54                 $\pm$ 0.11 & 99.86                 $\pm$ 0.03 & 99.37                 $\pm$ 0.06 & 99.77                 $\pm$ 0.02 & 99.10                 $\pm$ 0.40 \\
 11 & 91.44 $\pm$ 6.62 & 88.87$\downarrow$     $\pm$ 7.26 & 93.25                 $\pm$ 2.89 & 90.53                 $\pm$ 1.44 & 92.41                 $\pm$ 4.96 & 91.10                 $\pm$ 3.76 \\
 12 & 91.33 $\pm$ 2.04 & 91.33                 $\pm$ 7.02 & {\bf 95.08$\uparrow$} $\pm$ 2.08 & 94.40                 $\pm$ 0.94 & 91.77                 $\pm$ 6.33 & 91.75                 $\pm$ 2.28 \\
 13 & 80.87 $\pm$ 1.32 & 79.74                 $\pm$ 1.74 & 83.81                 $\pm$ 6.59 & {\bf 84.23$\uparrow$} $\pm$ 7.59 & 81.41                 $\pm$ 0.76 & 80.86                 $\pm$ 7.26 \\
 14 & 72.28 $\pm$ 2.81 & 76.60                 $\pm$ 3.10 & {\bf 77.59$\uparrow$} $\pm$ 2.84 & {\bf 78.35$\uparrow$} $\pm$ 5.11 & {\bf 80.40$\uparrow$} $\pm$ 3.31 & {\bf 77.74$\uparrow$} $\pm$ 1.69 \\
 15 & 98.01 $\pm$ 0.60 & 76.40$\downarrow$     $\pm$ 3.75 & 77.59$\downarrow$     $\pm$ 1.81 & 77.94$\downarrow$     $\pm$ 0.63 & 80.26$\downarrow$     $\pm$ 7.99 & 77.38$\downarrow$     $\pm$ 4.97 \\
 16 & 94.24 $\pm$ 0.39 & 95.63                 $\pm$ 1.83 & {\bf 97.67$\uparrow$} $\pm$ 1.62 & {\bf 99.09$\uparrow$} $\pm$ 0.14 & {\bf 99.85$\uparrow$} $\pm$ 0.04 & {\bf 97.62$\uparrow$} $\pm$ 1.05 \\\hline
 17 & 74.71 $\pm$ 8.62 & 72.49                 $\pm$ 0.48 & 72.55                 $\pm$ 9.85 & {\bf 77.08$\uparrow$} $\pm$ 8.89 & 76.90                 $\pm$ 0.86 & 72.51                 $\pm$ 5.30 \\
 18 & 89.82 $\pm$ 5.33 & 70.33$\downarrow$     $\pm$ 7.94 & 74.21$\downarrow$     $\pm$ 5.66 & 70.37$\downarrow$     $\pm$ 7.81 & 70.56$\downarrow$     $\pm$ 7.49 & 72.48$\downarrow$     $\pm$ 1.62 \\
 19 & 60.60 $\pm$ 1.83 & {\bf 68.13$\uparrow$} $\pm$ 3.76 & {\bf 72.70$\uparrow$} $\pm$ 8.05 & {\bf 71.12$\uparrow$} $\pm$ 5.40 & {\bf 71.46$\uparrow$} $\pm$ 7.62 & {\bf 69.32$\uparrow$} $\pm$ 0.08 \\
 20 & 58.88 $\pm$ 5.70 & 17.84$\downarrow$     $\pm$ 2.33 & 22.47$\downarrow$     $\pm$ 1.06 & 19.84$\downarrow$     $\pm$ 1.48 & 21.98$\downarrow$     $\pm$ 1.86 & 19.49$\downarrow$     $\pm$ 1.18 \\
 21 & 88.73 $\pm$ 3.01 & 69.68$\downarrow$     $\pm$ 4.14 & 70.71$\downarrow$     $\pm$ 9.94 & 72.29$\downarrow$     $\pm$ 8.70 & 74.23$\downarrow$     $\pm$ 1.22 & 71.04$\downarrow$     $\pm$ 7.45 \\
 22 & 69.18 $\pm$ 8.92 & {\bf 74.26$\uparrow$} $\pm$ 1.32 & {\bf 77.71$\uparrow$} $\pm$ 9.31 & {\bf 79.11$\uparrow$} $\pm$ 1.26 & {\bf 78.56$\uparrow$} $\pm$ 9.60 & {\bf 75.70$\uparrow$} $\pm$ 7.67 \\
 23 & 89.16 $\pm$ 1.33 & 90.90                 $\pm$ 1.18 & 91.08                 $\pm$ 5.23 & {\bf 95.12$\uparrow$} $\pm$ 1.01 & {\bf 93.26$\uparrow$} $\pm$ 0.67 & 91.32                 $\pm$ 2.97 \\\hline
Average & {\bf 81.61} $\pm$ 11.66
        & 78.12 $\pm$ 16.28
        & 80.92 $\pm$ 16.04
        & 81.02 $\pm$ 16.44
        & 81.07 $\pm$ 15.78
        & 79.36 $\pm$ 16.24 \\
$\bullet$ significance -- $p$ value                 &  &   0.150       &  0.771       &  0.812        &  0.818        &  0.344       \\
$\bullet$ win/loss/draw  &  & 10/12/1  & 17/6/0  & 18/5/0  & 19/4/0  & 15/8/0  \\
$\bullet$ sig.win/sig.loss &  & 3/5 & 9/4 & 11/4 & 10/4 & 7/4 \\\hline
\end{tabular}
\caption{Average rule set accuracy with standard deviation ({\em
ACC $\pm$ sd}) for different variants of the unordered algorithm using
  10-fold stratified cross-validation.}
\label{tab-results-Accuracy-Unordered}
}
%\end{small}
\end{center}
%\vspace*{-1cm}
\end{table}
\setlength{\tabcolsep}{1.4pt}

%------------------------------------------------------------------------------
%\begin{onecolumn}
%begin{table}[!ht]
\begin{table}[p]
\begin{center}
\vspace*{0.3cm}
{\footnotesize
%\begin{small}
\begin{tabular}{lrrrrrr}
\hline\noalign{\smallskip}
%\tiny
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD}
          & \multicolumn{1}{c}{\bf CN2-SD} \\
\#        & \multicolumn{1}{c}{\bf standard}
          & \multicolumn{1}{c}{\bf WRAcc}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
          & \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
          & \multicolumn{1}{c}{\bf (add.)} \\
          & \multicolumn{1}{c}{\em AUC $\pm$ sd} &
            \multicolumn{1}{c}{\em AUC $\pm$ sd} &
            \multicolumn{1}{c}{\em AUC $\pm$ sd} &
            \multicolumn{1}{c}{\em AUC $\pm$ sd} &
            \multicolumn{1}{c}{\em AUC $\pm$ sd} &
            \multicolumn{1}{c}{\em AUC $\pm$ sd} \\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
  1 & 33.39 $\pm$ 5.61 & {\bf 86.12$\uparrow$} $\pm$ 0.05 & {\bf 83.31$\uparrow$} $\pm$ 2.01 & {\bf 84.27$\uparrow$} $\pm$ 9.44 & {\bf 84.47$\uparrow$} $\pm$ 6.05 & {\bf 85.12$\uparrow$} $\pm$ 5.16 \\
  2 & 90.74 $\pm$ 3.57 & 89.52                 $\pm$ 7.26 & {\bf 94.37$\uparrow$} $\pm$ 2.29 & {\bf 96.28$\uparrow$} $\pm$ 1.47 & {\bf 97.33$\uparrow$} $\pm$ 0.98 & {\bf 94.52$\uparrow$} $\pm$ 1.67 \\
  3 & 84.51 $\pm$ 0.15 & 80.11$\downarrow$     $\pm$ 9.84 & 82.58                 $\pm$ 5.60 & 80.98$\downarrow$     $\pm$ 8.12 & 78.38$\downarrow$     $\pm$ 7.44 & 83.03                 $\pm$ 1.88 \\
  4 & 96.22 $\pm$ 2.55 & 93.59                 $\pm$ 2.26 & 97.19                 $\pm$ 0.76 & 92.37$\downarrow$     $\pm$ 2.33 & 96.54                 $\pm$ 1.90 & 92.87$\downarrow$     $\pm$ 2.66 \\
  5 & 71.33 $\pm$ 7.86 & {\bf 80.75$\uparrow$} $\pm$ 0.51 & {\bf 80.52$\uparrow$} $\pm$ 1.82 & {\bf 80.56$\uparrow$} $\pm$ 8.17 & {\bf 80.76$\uparrow$} $\pm$ 5.02 & {\bf 80.06$\uparrow$} $\pm$ 3.49 \\
  6 & 70.53 $\pm$ 5.99 & 64.42$\downarrow$     $\pm$ 3.29 & 68.09                 $\pm$ 7.34 & 68.63                 $\pm$ 2.44 & 64.02$\downarrow$     $\pm$ 8.71 & 70.61                 $\pm$ 2.46 \\
  7 & 71.99 $\pm$ 5.76 & 74.00                 $\pm$ 7.19 & 73.99                 $\pm$ 7.63 & 73.92                 $\pm$ 6.01 & {\bf 75.29$\uparrow$} $\pm$ 7.70 & 72.73                 $\pm$ 3.84 \\
  8 & 74.17 $\pm$ 5.35 & 73.98                 $\pm$ 0.90 & {\bf 83.82$\uparrow$} $\pm$ 9.76 & {\bf 84.69$\uparrow$} $\pm$ 0.63 & {\bf 87.02$\uparrow$} $\pm$ 9.80 & {\bf 85.62$\uparrow$} $\pm$ 1.84 \\
  9 & 78.81 $\pm$ 4.64 & {\bf 85.65$\uparrow$} $\pm$ 0.33 & {\bf 84.82$\uparrow$} $\pm$ 2.78 & {\bf 82.80$\uparrow$} $\pm$ 5.19 & 78.66                 $\pm$ 6.12 & {\bf 81.29$\uparrow$} $\pm$ 0.23 \\
 10 & 96.22 $\pm$ 2.31 & {\bf 98.59$\uparrow$} $\pm$ 0.10 & 97.13                 $\pm$ 0.78 & 96.54                 $\pm$ 0.13 & {\bf 99.65$\uparrow$} $\pm$ 0.04 & 97.42                 $\pm$ 0.24 \\
 11 & 94.46 $\pm$ 1.52 & 90.86$\downarrow$     $\pm$ 0.32 & 93.17                 $\pm$ 2.68 & 93.99                 $\pm$ 2.83 & 94.30                 $\pm$ 2.10 & 93.87                 $\pm$ 1.07 \\
 12 & 99.17 $\pm$ 0.23 & 99.17                 $\pm$ 0.16 & 99.96                 $\pm$ 0.01 & 99.38                 $\pm$ 0.15 & 99.92                 $\pm$ 0.03 & 99.46                 $\pm$ 0.06 \\
 13 & 83.20 $\pm$ 8.68 & 78.38$\downarrow$     $\pm$ 2.33 & 82.11                 $\pm$ 1.04 & 84.74                 $\pm$ 4.51 & 80.12$\downarrow$     $\pm$ 4.12 & 83.06                 $\pm$ 6.97 \\
 14 & 75.06 $\pm$ 6.13 & {\bf 79.41$\uparrow$} $\pm$ 5.12 & {\bf 81.62$\uparrow$} $\pm$ 7.61 & {\bf 79.97$\uparrow$} $\pm$ 1.29 & {\bf 80.12$\uparrow$} $\pm$ 5.34 & {\bf 78.51$\uparrow$} $\pm$ 1.15 \\
 15 & 97.90 $\pm$ 0.36 & 78.90$\downarrow$     $\pm$ 6.95 & 91.88$\downarrow$     $\pm$ 2.73 & 91.28$\downarrow$     $\pm$ 2.63 & 90.87$\downarrow$     $\pm$ 2.01 & 89.15$\downarrow$     $\pm$ 4.32 \\
 16 & 96.88 $\pm$ 1.67 & 96.41                 $\pm$ 1.63 & 93.44$\downarrow$     $\pm$ 2.97 & 95.35                 $\pm$ 0.18 & 94.82                 $\pm$ 1.06 & 93.95$\downarrow$     $\pm$ 2.06 \\\hline
Average & 82.16 $\pm$ 16.81
        & 84.37 $\pm$  9.87
        & {\bf 86.75} $\pm$  8.95
        & 86.61 $\pm$  8.81
        & 86.39 $\pm$ 10.32
        & 86.33 $\pm$  8.60 \\
$\bullet$ significance -- $p$ value                 &  & 0.563       & {\bf  0.175}       &  0.198       & 0.236       &  0.236       \\
$\bullet$ win/loss/draw  &  & 6/9/1  & {\bf 10/6/0} & 10/6/0  & 9/7/0  & 10/6/0  \\
$\bullet$ sig.win/sig.loss &  & 5/5 & {\bf 6/2} & 6/3 & 7/4 & 6/3 \\\hline
\end{tabular}
\caption{Area under the ROC curve (AUC-Method-2) with standard deviation ({\em AUC
$\pm$ sd}) for different variants of the unordered algorithm using
  10-fold stratified cross-validation.}
\label{tab-results-AUC-Unordered}
}
%\end{small}
\end{center}
%\vspace*{-1cm}
\end{table}
\setlength{\tabcolsep}{1.4pt}

\clearpage

\vskip 0.2in
\bibliography{lavrac04a}
\bibliographystyle{natbib}


%\begin{thebibliography}{99}
%%
%\bibitem{Agrawaletal.1996}
%  R. Agrawal, H. Mannila, R. Srikant, H. Toivonen, and A.I. Verkamo.
%  Fast discovery of association rules. In U.M. Fayyad, G.
%  Piatetski-Shapiro, P. Smyth and R.  Uthurusamy, editors,  {\it Advances
%    in Knowledge Discovery and Data
%      Mining},  307--328. AAAI Press, 1996.
%%
%\bibitem{ces:1990} B.~Cestnik.
%  Estimating probabilities: A crucial task in machine learning. In
%  L.~Aiello, editor, {\em Proceedings of the Ninth European Conference on
%    Artificial Intelligence},  147--149.
%Pitman,  1990.
%%
%\bibitem{cla:bos:1991}
%  P. Clark and R. Boswell. Rule induction with CN2: Some recent
%  improvements. In Y.~Kodratoff, editor, {\em Proceedings of the Fifth
%    European Working Session on  Learning}, 151--163. Springer, 1991.
%%
%\bibitem{cla:nib:1989}
%P. Clark and T. Niblett. The CN2 induction algorithm.
%{\em Machine Learning},  3(4): 261--283, 1989.
%%
%\bibitem{cohen1995}
%  W.W. Cohen. (1995) Fast effective rule induction.  In {\em Proceedings  of
%    the Twelfth International Conference on Machine Learning},
%  115--123.  Morgan Kaufmann, 1995.
%%
%\bibitem{cohen1999} W.W. Cohen and Y. Singer.  A simple, fast, and
%  effective rule learner.  In {\em Proceedings of AAAI/IAAI}, 335--342.
%AAAI Press, 1999.
%%
%\bibitem{DeRaedtandBlockeel1997}
%L. De Raedt and H. Blockeel. Using logical decision trees for clustering
%In {\em Proceedings of the Seventh International Workshop on Inductive Logic
%Programming}, 133--140. Springer, 1997.
%%
%\bibitem{DeRaedt2001}
%L. De Raedt, H. Blockeel, L. Dehaspe and W. Van Laer.
%Three companions for data mining in first order logic.
%In S. D\v zeroski and N. Lavra\v c, editors, {\em Relational Data
%  Mining}, 106--139. Springer, 2001.
%%
%\bibitem{DeRaedt1997}
%L. De Raedt and L. Dehaspe. Clausal discovery. {\em Machine
%Learning} 26: 99--146, 1997.
%%
%\bibitem{Dzeroskietal.1993}
%  S. D\v{z}eroski, B. Cestnik, and I. Petrovski. (1993) \newblock
%  Using the m-estimate in rule induction.  \newblock {\em Journal of
%    Computing and Information Technology}, 1(1):37 -- 46, 1993.
%%
%\bibitem{fay:ira:1993}
%  U.M. Fayyad and K.B. Irani, K.B.  Multi-interval discretisation of
%  continuous-valued attributes for classification learning. In R.
%  Bajcsy, editor, {\em Proceedings of the Thirteenth International Joint
%    Conference on Artificial Intelligence}, 1022--1027. Morgan
%  Kaufmann, 1993.
%%
%\bibitem{Ferrietal.2002}
%  C. Ferri-Ram\'{\i}rez, P.A. Flach, and J. Hernandez-Orallo.  Learning
%  decision trees using the area under the ROC curve. In {\em Proceedings of
%    the Nineteenth International Conference on Machine Learning}, 139--146.
%  Morgan Kaufmann, 2002.
%%
%\bibitem{flach-icml03}
%P. Flach.  The geometry of ROC space:
%  Understanding machine learning metrics through ROC isometrics. In
%  {\em Proceedings of the Twentieth International Conference
%    on Machine Learning}, 194--201. AAAI Press, 2003.
%%
%\bibitem{FlachandLavrac2003}
%P. Flach and N. Lavra\v{c}. Rule induction.  In M. Berthold and
%D.J. Hand, editors, {\em Intelligent Data Analysis}, 229--267.
%Springer 2003.
%%
%\bibitem{FlachandSavnik1999}
%P. Flach and I. Savnik. Database dependency discovery: A machine
%learning approach. {\em AI Communications}, 12(3): 139-160, 1999.
%%
%\bibitem{FreundandShapire1996}
%  Y. Freund and R.E. Shapire. Experiments with a new boosting
%  algorithm. In {\em Proceedings of the Thirteenth International Conference on
%    Machine Learning}, 148--156. Morgan Kaufmann, 1996.
%%
%%\bibitem{GambergerLavrac2000}
%%D. Gamberger and N.Lavra\v{c}.
%%  Confirmation rule sets.  In {\em Proceedings of the Fourth European
%%    Conference on Principles of Data Mining and Knowledge Discovery},
%%  pp. 34--43, Springer, 2000.
%%
%%
%\bibitem{FriedmanandFisher1999}
%  J.H. Friedman and N.I. Fisher. Bump hunting in high-dimensional
%  data. {\em Statistics and Computing}, 9: 123--143, 1999.
%%
%\bibitem{furnkranz-flach-icml03}
%J. F\"{u}rnkranz and P.A. Flach. An analysis of rule evaluation
%metrics. In {\em Proceedings of the Twetieth International
%Conference on Machine Learning}, 202--209. AAAI Press, 2003.
%%
%\bibitem{GambergerLavrac2002}
%  D. Gamberger and N. Lavra\v{c}.  Expert guided subgroup
%  discovery: Methodology and application. {\it Journal of
%  Artificial Intelligence Research}, 17: 501--527,  2002.
%%
%\bibitem{Gordon1982}
%A.D. Gordon. {\em Classification}. London: Chapman and
%Hall, 1982.
%%
%\bibitem{GowdaandDiday1992}
%K.C. Gowda, K. and E. Diday. Symbolic clustering using a new
%dissimilarity measure. {\it IEEE Transactions on Systems, Man, and
%Cybernetics}, 22(2): 567--578, 1992.
%%
%\bibitem{HandandTill2001}
%D.J. Hand and R.J. Till. A simple generalisation of the area under the
%ROC curve for multiple class classification problems. {\em Machine
%Learning}, 45: 171--186, 2001.
%%
%\bibitem{Hsu1998}
%  D. Hsu, O. Etzioni and S. Soderland.  A redundant covering algorithm
%  applied to text classification.  In {\em Proceedings of the AAAI Workshop
%  on Learning from Text Categorization}. AAAI Press, 1998.
%%
%\bibitem{JovanoskiandLavrac2001} V. Jovanoski and N. Lavra\v{c}.
%  Classification rule learning with APRIORI-C.  In {\em Progress in
%    Artificial Intelligence: Proceedings of the Tenth Portuguese
%    Conference on
%    Artificial Intelligence}, 44--51. Springer, 2001.
%%
%\bibitem{Kavseketal.2003}
%B. Kav\v{s}ek, N. Lavra\v{c} and V. Jovanoski. APRIORI-SD:
%Adapting association rule learning to subgroup discovery. In {\em
%Proceedings of the Fifth International Symposium on Intelligent Data
%Analysis}, 230--241. Springer, 2003.
%%
%\bibitem{Klosgen1996}
%W. Kl\"{o}sgen.
%Explora: A multipattern
%  and multistrategy discovery assistant.  In U.M. Fayyad, G.
%  Piatetski-Shapiro, P. Smyth and  R.  Uthurusamy, editors, {\em Advances
%    in Knowledge Discovery and Data Mining}, 249--271.  MIT Press, 1996.
%%
%\bibitem{Knobbeetal.2001}
%A.J. Knobbe, M. de Haas and A. Siebes. Propositionalisation and
%Aggregates. In {\em Proceedings of the Fifth European Conference on
%Principles and Practice of Knowledge Discovery in Databases},
%277--288. Springer, 2001.
%%
%\bibitem{Knobbeetal.2002}
%A.J. Knobbe, A. Siebes and B. Marseille. Involving aggregate
%functions in multi-relational search. In {\em Proceedings of the Sixth
%European Conference on Principles and Practice of Knowledge
%Discovery in Databases}, 287--298. Springer, 2002.
%%
%\bibitem{Langley1996}
%P. Langley. {\em Elements of Machine Learning}. Morgan
%Kaufmann, 1996.
%%
%\bibitem{Lavracetal.2002}
%N.  Lavra\v c, P. Flach, B. Kav\v sek and L. Todorovski (2002)
%  Adapting classification rule induction to subgroup discovery. In
%  {\em Proceedings of the Second IEEE International Conference on Data
%    Mining}, 266--273. IEEE Computer Society, 2002.
%%
%\bibitem{lav:fla:zup:1999} N. Lavra\v{c}, P. Flach and B. Zupan.
%  \newblock Rule evaluation measures: {A} unifying view.  \newblock In
%  {\em Proceedings of the Nineth International Workshop on Inductive Logic
%    Programming}, 74--185. Springer, 1999.
%%
%\bibitem{Lavracetal.2002a}
%N.  Lavra\v c, F.  \v{Z}elezn\'{y} and P. Flach.
%  RSD: Relational subgroup discovery through first-order feature
%  construction. In {\em Proceedings of the Twelfth
%  International Conference on
%  Inductive Logic Programming}, 149--165.  Springer, 2003.
%%
%\bibitem{Leeetal.1998}
%  Y. Lee, B.G. Buchanan, and J.M.  Aronis.  Knowledge-based learning
%  in exploratory science: {L}earning rules to
%predict rodent carcinogenicity. {\em Machine Learning}, 30: 217--240, 1998.%
%%
%\bibitem{Ljubic:2002}
%P. Ljubi\v{c}, L. Todorovski, N. Lavra\v{c}, and J.C. Bullas.
%Time-series analysis of UK traffic accident data. In {\em Proceedings of
%the Fifth International Multi-conference Information Society},
%131--134, 2002.
%%
%\bibitem{Michalski1980}
%R.S. Michalski. Pattern recognition as rule-guided inductive
%inference. {\em IEEE Transactions on Pattern Analysis and Machine
%5Intelligence}, 2(4): 349--361, 1980.
%%
%\bibitem{Michalskietal.1986}
%R.S. Michalski, I. Mozeti\v c, J. Hong, and N. Lavra\v c.
%The multi-purpose incremental learning system {AQ15} and its testing
%    application on three medical domains.
%In {\em Proceedings of the Fifth National Conference on Artificial
%Intelligence},
%   1041--1045.  Morgan Kaufmann, 1986.
%%
%\bibitem{mur:aha:1994}
%P.M. Murphy and D.W. Aha. UCI repository of
%   machine learning databases. [http://www.ics.uci.edu/\~{
%    }mlearn/MLRepository.html]. Irvine, CA: University of California,
%Department of Information and Computer Science, 1994.
%%
%\bibitem{ProvostandFawcett2001}
%F. Provost and T. Fawcett.
%Robust classification for imprecise environments. {\em Machine Learning},
%42(3): 203--231, 2001.
%%
%\bibitem{riv:1987}
%R.L. Rivest. Learning decision lists. {\em Machine Learning},
%2(3): 229--246, 1987.
%%
%\bibitem{SchapireandSinger1998}
%  R.E. Schapire and Y.  Singer.  Improved boosting algorithms using
%  confidence-rated predictions. In {\em Proceedings of the Eleventh Conference
%    on Computational Learning Theory}, 80--91. ACM Press, 1998.
%%
%\bibitem{Silberschatz95}
%A. Silberschatz and A. Tuzhilin.
%On subjective measures of interestingness in knowledge discovery.
%In {\em Proceedigns of the First International Conference on Knowledge Discovery
%and Data Mining}, 275--281, 1995.
%%
%\bibitem{SokalandSneath1963}
%R.R.Slokal and  P.H.A. Sneath. {\em Principles of
%Numerical Taxonomy}. San Francisco: Freeman, 1963.
%%
%\bibitem{tod:fla:lav:2000}
%L. Todorovski, P. Flach, and N. Lavra\v c.  Predictive
%performance of weighted relative accuracy. In D.A. Zighed, J.
%Komorowski,  and J. Zytkow, editors, {\em Proceedings of the
%Fourth European Conference on Principles of Data Mining and Knowledge
%Discovery}, 255--264. Springer, 2000.
%%
%\bibitem{wit:fra:1999}
%I.H. Witten and E. Frank. {\em Data Mining: Practical
%Machine Learning Tools and Techniques with Java Implementations}.
%Morgan Kaufmann, 1999.
%%
%\bibitem{Wrobel1997}
%  S. Wrobel.  An algorithm for multi-relational discovery of
%  subgroups. In {\em Proceedings of the First European Conference on
%  Principles of Data Mining and Knowledge Discovery}, 78--87. Springer, 1997.
%%
%\bibitem{Wrobel2001}
%S. Wrobel. Inductive logic programming for knowledge discovery in
%databases. In S. D\v zeroski and N. Lavra\v c, editors, {\em
%Relational Data
%  Mining}, 74--101. Springer, 2001.
%
%\end{thebibliography}

\end{document}


%\newpage
%
%\appendix
%\section*{Appendix B.}
%\subsection*{Tables with detailed results for different variants of the ordered algorithm in UCI data sets}
%\label{app-ordered}
%
%%\subsection{UCI domains}
%
%%---------------------------------------------------------------------
%%\begin{onecolumn}
%%begin{table}[!ht]
%\begin{table}[!h]
%\begin{center}
%\caption{Area under the ROC curve (AUC-Method-2) with standard deviation ({\em AUC
%  $\pm$ sd}) for different variants of the ordered algorithm using
%  10-fold stratified cross-validation.}
%\label{tab-results-AUC-Ordered}
%\vspace*{0.3cm}
%{\footnotesize
%%\begin{small}
%\begin{tabular}{crrrrrr}
%\hline\noalign{\smallskip}
%\hline\noalign{\smallskip}
%%\tiny
%          & \multicolumn{1}{c}{\bf CN2}
%          & \multicolumn{1}{c}{\bf CN2}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD} \\
%    \#    & \multicolumn{1}{c}{\bf standard}
%          & \multicolumn{1}{c}{\bf WRAcc}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
%          & \multicolumn{1}{c}{\bf (add.)} \\
%          & \multicolumn{1}{c}{\em AUC $\pm$ sd} &
%            \multicolumn{1}{c}{\em AUC $\pm$ sd} &
%            \multicolumn{1}{c}{\em AUC $\pm$ sd} &
%            \multicolumn{1}{c}{\em AUC $\pm$ sd} &
%            \multicolumn{1}{c}{\em AUC $\pm$ sd} &
%            \multicolumn{1}{c}{\em AUC $\pm$ sd} \\
%
%\noalign{\smallskip}
%\hline
%\noalign{\smallskip}
%  1 & 85.81 $\pm$ 0.01 & 84.12 $\pm$ 0.00 & 86.65 $\pm$ 0.03 & 86.73 $\pm$ 0.03 & 86.20 $\pm$ 0.03 & 86.23 $\pm$ 0.03 \\
%  2 & 91.36 $\pm$ 0.10 & 91.90 $\pm$ 0.04 & 91.57 $\pm$ 0.05 & 91.18 $\pm$ 0.04 & 93.89 $\pm$ 0.05 & 91.23 $\pm$ 0.04 \\
%  3 & 85.66 $\pm$ 0.03 & 86.00 $\pm$ 0.03 & 85.31 $\pm$ 0.02 & 81.30 $\pm$ 0.02 & 82.40 $\pm$ 0.01 & 83.68 $\pm$ 0.02 \\
%  4 & 91.90 $\pm$ 0.01 & 91.36 $\pm$ 0.03 & 91.49 $\pm$ 0.04 & 91.46 $\pm$ 0.04 & 91.33 $\pm$ 0.04 & 93.41 $\pm$ 0.04 \\
%  5 & 78.41 $\pm$ 0.08 & 76.97 $\pm$ 0.04 & 78.04 $\pm$ 0.06 & 78.78 $\pm$ 0.05 & 78.37 $\pm$ 0.05 & 77.45 $\pm$ 0.04 \\
%  6 & 65.67 $\pm$ 0.06 & 67.97 $\pm$ 0.03 & 64.35 $\pm$ 0.05 & 68.81 $\pm$ 0.06 & 68.06 $\pm$ 0.06 & 69.27 $\pm$ 0.05 \\
%  7 & 68.16 $\pm$ 0.08 & 72.10 $\pm$ 0.11 & 74.19 $\pm$ 0.14 & 73.89 $\pm$ 0.13 & 74.45 $\pm$ 0.13 & 71.25 $\pm$ 0.12 \\
%  8 & 75.67 $\pm$ 0.09 & 76.35 $\pm$ 0.07 & 87.58 $\pm$ 0.09 & 86.90 $\pm$ 0.12 & 85.79 $\pm$ 0.11 & 86.52 $\pm$ 0.09 \\
%  9 & 80.53 $\pm$ 0.04 & 79.84 $\pm$ 0.06 & 83.69 $\pm$ 0.04 & 85.10 $\pm$ 0.04 & 82.96 $\pm$ 0.04 & 85.60 $\pm$ 0.03 \\
% 10 & 96.93 $\pm$ 0.00 & 99.30 $\pm$ 0.00 & 98.00 $\pm$ 0.00 & 97.24 $\pm$ 0.00 & 98.65 $\pm$ 0.00 & 94.81 $\pm$ 0.00 \\
% 11 & 89.13 $\pm$ 0.05 & 76.18 $\pm$ 0.08 & 90.00 $\pm$ 0.05 & 91.94 $\pm$ 0.04 & 92.01 $\pm$ 0.04 & 90.14 $\pm$ 0.06 \\
% 12 & 92.73 $\pm$ 0.01 & 92.73 $\pm$ 0.01 & 92.56 $\pm$ 0.00 & 92.56 $\pm$ 0.00 & 92.56 $\pm$ 0.00 & 92.56 $\pm$ 0.01 \\
% 13 & 76.28 $\pm$ 0.11 & 78.74 $\pm$ 0.09 & 78.68 $\pm$ 0.15 & 80.37 $\pm$ 0.16 & 79.89 $\pm$ 0.18 & 79.10 $\pm$ 0.17 \\
% 14 & 75.02 $\pm$ 0.01 & 80.43 $\pm$ 0.07 & 74.79 $\pm$ 0.02 & 76.06 $\pm$ 0.03 & 80.07 $\pm$ 0.03 & 73.36 $\pm$ 0.03 \\
% 15 & 66.28 $\pm$ 0.01 & 65.32 $\pm$ 0.07 & 76.89 $\pm$ 0.02 & 77.71 $\pm$ 0.03 & 74.95 $\pm$ 0.03 & 68.01 $\pm$ 0.03 \\
% 16 & 90.73 $\pm$ 0.01 & 97.18 $\pm$ 0.07 & 96.14 $\pm$ 0.02 & 93.22 $\pm$ 0.03 & 93.59 $\pm$ 0.03 & 97.99 $\pm$ 0.03 \\\hline
%Avg & {\bf 81.89 $\pm$ 0.05}
%    & {\bf 82.28 $\pm$ 0.05}
%    & {\bf 84.37 $\pm$ 0.05}
%    & {\bf 84.58 $\pm$ 0.05}
%    & {\bf 84.70 $\pm$ 0.05}
%    & {\bf 83.79 $\pm$ 0.05} \\
%\hline
%\end{tabular}
%}
%%\end{small}
%\end{center}
%%\vspace*{-1cm}
%\end{table}
%\setlength{\tabcolsep}{1.4pt}
%
%%-------------------------------------------------------------------------------------------------------------
%%\begin{onecolumn}
%%begin{table}[!ht]
%\begin{table}[!h]
%\begin{center}
%\caption{Sizes of rulesets with standard deviation ({\em S
%  $\pm$ sd}) for different variants of the ordered algorithm using
%  10-fold stratified cross-validation.}
%\label{tab-results-Size-Ordered}
%\vspace*{0.3cm}
%{\footnotesize
%%\begin{small}
%\begin{tabular}{crrrrrr}
%\hline\noalign{\smallskip}
%%\tiny
%          & \multicolumn{1}{c}{\bf CN2}
%          & \multicolumn{1}{c}{\bf CN2}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD} \\
%    \#    & \multicolumn{1}{c}{\bf standard}
%          & \multicolumn{1}{c}{\bf WRAcc}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
%          & \multicolumn{1}{c}{\bf (add.)} \\
%          & \multicolumn{1}{c}{\em S $\pm$ sd} &
%            \multicolumn{1}{c}{\em S $\pm$ sd} &
%            \multicolumn{1}{c}{\em S $\pm$ sd} &
%            \multicolumn{1}{c}{\em S $\pm$ sd} &
%            \multicolumn{1}{c}{\em S $\pm$ sd} &
%            \multicolumn{1}{c}{\em S $\pm$ sd} \\
%
%\noalign{\smallskip}
%\hline
%\noalign{\smallskip}
%  1 & 13.4  $\pm$ 0.01 & 2.4  $\pm$ 0.00 & 2.9  $\pm$ 0.03 & 3.7  $\pm$ 0.03 & 4.1  $\pm$ 0.03 & 4.4  $\pm$ 0.03 \\
%  2 & 11.4  $\pm$ 0.10 & 5.9  $\pm$ 0.04 & 5.2  $\pm$ 0.05 & 4.3  $\pm$ 0.04 & 4.0  $\pm$ 0.05 & 5.0  $\pm$ 0.04 \\
%  3 & 1.3   $\pm$ 0.03 & 2.0  $\pm$ 0.03 & 2.6  $\pm$ 0.02 & 3.4  $\pm$ 0.02 & 3.2  $\pm$ 0.01 & 2.5  $\pm$ 0.02 \\
%  4 & 20.7  $\pm$ 0.01 & 2.0  $\pm$ 0.03 & 2.7  $\pm$ 0.04 & 2.5  $\pm$ 0.04 & 2.1  $\pm$ 0.04 & 1.5  $\pm$ 0.04 \\
%  5 & 12.3  $\pm$ 0.08 & 3.6  $\pm$ 0.04 & 4.5  $\pm$ 0.06 & 5.0  $\pm$ 0.05 & 5.6  $\pm$ 0.05 & 4.6  $\pm$ 0.04 \\
%  6 & 3.1   $\pm$ 0.06 & 1.6  $\pm$ 0.03 & 2.3  $\pm$ 0.05 & 2.0  $\pm$ 0.06 & 2.4  $\pm$ 0.06 & 1.5  $\pm$ 0.05 \\
%  7 & 15.5  $\pm$ 0.08 & 9.3  $\pm$ 0.11 & 9.3  $\pm$ 0.14 & 9.2  $\pm$ 0.13 & 9.3  $\pm$ 0.13 & 8.4  $\pm$ 0.12 \\
%  8 & 7.7   $\pm$ 0.09 & 3.7  $\pm$ 0.07 & 3.0  $\pm$ 0.09 & 2.4  $\pm$ 0.12 & 1.9  $\pm$ 0.11 & 1.6  $\pm$ 0.09 \\
%  9 & 3.1   $\pm$ 0.04 & 2.0  $\pm$ 0.06 & 2.7  $\pm$ 0.04 & 3.5  $\pm$ 0.04 & 3.8  $\pm$ 0.04 & 2.9  $\pm$ 0.03 \\
% 10 & 10.7  $\pm$ 0.00 & 2.0  $\pm$ 0.00 & 2.6  $\pm$ 0.00 & 1.8  $\pm$ 0.00 & 2.4  $\pm$ 0.00 & 1.8  $\pm$ 0.00 \\
% 11 & 7.2   $\pm$ 0.05 & 4.7  $\pm$ 0.08 & 4.9  $\pm$ 0.05 & 4.1  $\pm$ 0.04 & 4.7  $\pm$ 0.04 & 4.0  $\pm$ 0.06 \\
% 12 & 2.0   $\pm$ 0.01 & 2.0  $\pm$ 0.01 & 2.1  $\pm$ 0.00 & 2.0  $\pm$ 0.00 & 2.4  $\pm$ 0.00 & 2.6  $\pm$ 0.01 \\
% 13 & 4.6   $\pm$ 0.11 & 2.2  $\pm$ 0.09 & 1.7  $\pm$ 0.15 & 1.9  $\pm$ 0.16 & 2.6  $\pm$ 0.18 & 2.8  $\pm$ 0.17 \\
% 14 & 5.2   $\pm$ 0.01 & 2.3  $\pm$ 0.07 & 1.4  $\pm$ 0.02 & 1.3  $\pm$ 0.03 & 2.0  $\pm$ 0.03 & 1.3  $\pm$ 0.03 \\
% 15 & 30.8  $\pm$ 0.01 & 8.1  $\pm$ 0.07 & 7.7  $\pm$ 0.02 & 8.6  $\pm$ 0.03 & 9.0  $\pm$ 0.03 & 8.9  $\pm$ 0.03 \\
% 16 & 5.9   $\pm$ 0.01 & 1.5  $\pm$ 0.07 & 1.9  $\pm$ 0.02 & 2.8  $\pm$ 0.03 & 3.4  $\pm$ 0.03 & 2.4  $\pm$ 0.03 \\\hline
% 17 & 25.9  $\pm$ 0.00 & 4.5  $\pm$ 0.00 & 4.9  $\pm$ 0.00 & 5.6  $\pm$ 0.00 & 6.4  $\pm$ 0.00 & 4.7  $\pm$ 0.00 \\
% 18 & 34.5  $\pm$ 0.05 & 6.6  $\pm$ 0.08 & 6.4  $\pm$ 0.05 & 6.9  $\pm$ 0.04 & 6.2  $\pm$ 0.04 & 7.1  $\pm$ 0.06 \\
% 19 & 8.5   $\pm$ 0.01 & 3.0  $\pm$ 0.01 & 2.9  $\pm$ 0.00 & 3.8  $\pm$ 0.00 & 4.5  $\pm$ 0.00 & 3.6  $\pm$ 0.01 \\
% 20 & 23.1  $\pm$ 0.11 & 7.1  $\pm$ 0.09 & 8.0  $\pm$ 0.15 & 7.8  $\pm$ 0.16 & 8.7  $\pm$ 0.18 & 7.4  $\pm$ 0.17 \\
% 21 & 21.7  $\pm$ 0.01 & 11.1 $\pm$ 0.07 & 10.4 $\pm$ 0.02 & 10.9 $\pm$ 0.03 & 10.4 $\pm$ 0.03 & 11.2 $\pm$ 0.03 \\
% 22 & 139.1 $\pm$ 0.01 & 5.4  $\pm$ 0.07 & 6.4  $\pm$ 0.02 & 6.1  $\pm$ 0.03 & 5.5  $\pm$ 0.03 & 6.5  $\pm$ 0.03 \\
% 23 & 3.2   $\pm$ 0.01 & 2.0  $\pm$ 0.07 & 2.3  $\pm$ 0.02 & 1.5  $\pm$ 0.03 & 1.5  $\pm$ 0.03 & 1.2  $\pm$ 0.03 \\\hline
%Avg & {\bf 17.87 $\pm$ 0.05}
%    & {\bf  4.13 $\pm$ 0.05}
%    & {\bf  4.30 $\pm$ 0.05}
%    & {\bf  4.40 $\pm$ 0.05}
%    & {\bf  4.61 $\pm$ 0.05}
%    & {\bf  4.27 $\pm$ 0.05} \\
%\hline
%\end{tabular}
%}
%%\end{small}
%\end{center}
%%\vspace*{-1cm}
%\end{table}
%\setlength{\tabcolsep}{1.4pt}
%
%%-------------------------------------------------------------------------------------------------------------
%%\begin{onecolumn}
%%begin{table}[!ht]
%\begin{table}[!h]
%\begin{center}
%\caption{Average coverage per rule with standard deviation
%  ({\em COV $\pm$ sd}) for different variants of the ordered
%  algorithm using 10-fold stratified cross-validation.}
%\label{tab-results-Coverage-Ordered}
%\vspace*{0.3cm}
%{\footnotesize
%%\begin{small}
%\begin{tabular}{crrrrrr}
%\hline\noalign{\smallskip}
%%\tiny
%          & \multicolumn{1}{c}{\bf CN2}
%          & \multicolumn{1}{c}{\bf CN2}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD} \\
%    \#    & \multicolumn{1}{c}{\bf standard}
%          & \multicolumn{1}{c}{\bf WRAcc}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
%          & \multicolumn{1}{c}{\bf (add.)} \\
%          & \multicolumn{1}{c}{\em COV $\pm$ sd} &
%            \multicolumn{1}{c}{\em COV $\pm$ sd} &
%            \multicolumn{1}{c}{\em COV $\pm$ sd} &
%            \multicolumn{1}{c}{\em COV $\pm$ sd} &
%            \multicolumn{1}{c}{\em COV $\pm$ sd} &
%            \multicolumn{1}{c}{\em COV $\pm$ sd} \\
%
%\noalign{\smallskip}
%\hline
%\noalign{\smallskip}
%  1 & 56.0  $\pm$ 0.01 & 289.2  $\pm$ 0.00 & 304.9  $\pm$ 0.03 & 307.2  $\pm$ 0.03 & 322.3  $\pm$ 0.03 & 375.9  $\pm$ 0.03 \\
%  2 & 63.6  $\pm$ 0.10 & 120.2  $\pm$ 0.04 & 167.8  $\pm$ 0.05 & 124.9  $\pm$ 0.04 & 154.5  $\pm$ 0.05 & 207.1  $\pm$ 0.04 \\
%  3 & 79.1  $\pm$ 0.03 & 53.8   $\pm$ 0.03 & 84.0   $\pm$ 0.02 & 70.6   $\pm$ 0.02 & 91.1   $\pm$ 0.01 & 55.0   $\pm$ 0.02 \\
%  4 & 156.9 $\pm$ 0.01 & 1602.2 $\pm$ 0.03 & 1644.6 $\pm$ 0.04 & 1639.5 $\pm$ 0.04 & 1606.4 $\pm$ 0.04 & 1702.2 $\pm$ 0.04 \\
%  5 & 65.4  $\pm$ 0.08 & 217.3  $\pm$ 0.04 & 240.7  $\pm$ 0.06 & 222.1  $\pm$ 0.05 & 241.6  $\pm$ 0.05 & 217.6  $\pm$ 0.04 \\
%  6 & 45.3  $\pm$ 0.06 & 86.4   $\pm$ 0.03 & 124.9  $\pm$ 0.05 & 95.0   $\pm$ 0.06 & 88.2   $\pm$ 0.06 & 124.8  $\pm$ 0.05 \\
%  7 & 66.9  $\pm$ 0.08 & 108.2  $\pm$ 0.11 & 130.3  $\pm$ 0.14 & 145.3  $\pm$ 0.13 & 124.1  $\pm$ 0.13 & 206.0  $\pm$ 0.12 \\
%  8 & 39.9  $\pm$ 0.09 & 73.9   $\pm$ 0.07 & 98.7   $\pm$ 0.09 & 90.9   $\pm$ 0.12 & 107.3  $\pm$ 0.11 & 172.7  $\pm$ 0.09 \\
%  9 & 52.8  $\pm$ 0.04 & 80.2   $\pm$ 0.06 & 91.3   $\pm$ 0.04 & 103.2  $\pm$ 0.04 & 101.2  $\pm$ 0.04 & 172.4  $\pm$ 0.03 \\
% 10 & 299.7 $\pm$ 0.00 & 1585.8 $\pm$ 0.00 & 1597.5 $\pm$ 0.00 & 1606.8 $\pm$ 0.00 & 1619.9 $\pm$ 0.00 & 1642.7 $\pm$ 0.00 \\
% 11 & 50.9  $\pm$ 0.05 & 76.2   $\pm$ 0.08 & 123.7  $\pm$ 0.05 & 90.4   $\pm$ 0.04 & 100.0  $\pm$ 0.04 & 76.3   $\pm$ 0.06 \\
% 12 & 77.7  $\pm$ 0.01 & 77.1   $\pm$ 0.01 & 93.2   $\pm$ 0.00 & 92.8   $\pm$ 0.00 & 101.9  $\pm$ 0.00 & 176.3  $\pm$ 0.01 \\
% 13 & 42.2  $\pm$ 0.11 & 87.7   $\pm$ 0.09 & 131.3  $\pm$ 0.15 & 90.8   $\pm$ 0.16 & 101.8  $\pm$ 0.18 & 169.4  $\pm$ 0.17 \\
% 14 & 36.8  $\pm$ 0.01 & 83.0   $\pm$ 0.07 & 110.3  $\pm$ 0.02 & 86.5   $\pm$ 0.03 & 89.5   $\pm$ 0.03 & 156.0  $\pm$ 0.03 \\
% 15 & 33.1  $\pm$ 0.01 & 122.3  $\pm$ 0.07 & 166.1  $\pm$ 0.02 & 141.2  $\pm$ 0.03 & 149.1  $\pm$ 0.03 & 174.7  $\pm$ 0.03 \\
% 16 & 75.2  $\pm$ 0.01 & 292.7  $\pm$ 0.07 & 318.6  $\pm$ 0.02 & 332.6  $\pm$ 0.03 & 323.0  $\pm$ 0.03 & 299.1  $\pm$ 0.03 \\\hline
% 17 & 24.2  $\pm$ 0.00 & 141.1  $\pm$ 0.00 & 143.5  $\pm$ 0.00 & 142.7  $\pm$ 0.00 & 166.0  $\pm$ 0.00 & 236.2  $\pm$ 0.00 \\
% 18 & 53.7  $\pm$ 0.05 & 262.5  $\pm$ 0.08 & 300.6  $\pm$ 0.05 & 290.0  $\pm$ 0.04 & 273.8  $\pm$ 0.04 & 288.0  $\pm$ 0.06 \\
% 19 & 28.9  $\pm$ 0.01 & 72.9   $\pm$ 0.01 & 106.2  $\pm$ 0.00 & 105.1  $\pm$ 0.00 & 93.5   $\pm$ 0.00 & 172.3  $\pm$ 0.01 \\
% 20 & 100.9 $\pm$ 0.11 & 328.0  $\pm$ 0.09 & 332.6  $\pm$ 0.15 & 340.6  $\pm$ 0.16 & 361.3  $\pm$ 0.18 & 419.0  $\pm$ 0.17 \\
% 21 & 34.9  $\pm$ 0.01 & 66.4   $\pm$ 0.07 & 75.1   $\pm$ 0.02 & 104.6  $\pm$ 0.03 & 70.6   $\pm$ 0.03 & 129.6  $\pm$ 0.03 \\
% 22 & 39.4  $\pm$ 0.01 & 928.6  $\pm$ 0.07 & 929.7  $\pm$ 0.02 & 965.4  $\pm$ 0.03 & 965.1  $\pm$ 0.03 & 962.8  $\pm$ 0.03 \\
% 23 & 59.6  $\pm$ 0.01 & 92.4   $\pm$ 0.07 & 129.1  $\pm$ 0.02 & 96.5   $\pm$ 0.03 & 124.5  $\pm$ 0.03 & 106.2  $\pm$ 0.03 \\\hline
%Avg & {\bf  68.84 $\pm$ 0.05}
%    & {\bf 297.74 $\pm$ 0.05}
%    & {\bf 323.68 $\pm$ 0.05}
%    & {\bf 316.73 $\pm$ 0.05}
%    & {\bf 320.73 $\pm$ 0.05}
%    & {\bf 358.36 $\pm$ 0.05} \\
%\hline
%\end{tabular}
%}
%%\end{small}
%\end{center}
%%\vspace*{-1cm}
%\end{table}
%\setlength{\tabcolsep}{1.4pt}
%
%%-------------------------------------------------------------------------------------------------------------
%%\begin{onecolumn}
%%begin{table}[!ht]
%\begin{table}[!h]
%\begin{center}
%\caption{Average significance of a rule with standard deviation ({\em SIG
%  $\pm$ sd}) for different variants of the ordered algorithm using
%  10-fold stratified cross-validation.}
%\label{tab-results-Significance-Ordered}
%\vspace*{0.3cm}
%{\footnotesize
%%\begin{small}
%\begin{tabular}{crrrrrr}
%\hline\noalign{\smallskip}
%%\tiny
%          & \multicolumn{1}{c}{\bf CN2}
%          & \multicolumn{1}{c}{\bf CN2}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD} \\
%    \#    & \multicolumn{1}{c}{\bf standard}
%          & \multicolumn{1}{c}{\bf WRAcc}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
%          & \multicolumn{1}{c}{\bf (add.)} \\
%          & \multicolumn{1}{c}{\em SIG $\pm$ sd} &
%            \multicolumn{1}{c}{\em SIG $\pm$ sd} &
%            \multicolumn{1}{c}{\em SIG $\pm$ sd} &
%            \multicolumn{1}{c}{\em SIG $\pm$ sd} &
%            \multicolumn{1}{c}{\em SIG $\pm$ sd} &
%            \multicolumn{1}{c}{\em SIG $\pm$ sd} \\
%
%\noalign{\smallskip}
%\hline
%\noalign{\smallskip}
%  1 & 2.7 $\pm$ 0.01 & 14.3 $\pm$ 0.00 & 10.7 $\pm$ 0.03 & 22.8 $\pm$ 0.03 & 11.0 $\pm$ 0.03 & 6.8  $\pm$ 0.03 \\
%  2 & 1.2 $\pm$ 0.10 & 14.5 $\pm$ 0.04 & 9.2  $\pm$ 0.05 & 26.7 $\pm$ 0.04 & 1.3  $\pm$ 0.05 & 1.7  $\pm$ 0.04 \\
%  3 & 2.7 $\pm$ 0.03 & 4.8  $\pm$ 0.03 & 24.4 $\pm$ 0.02 & 16.1 $\pm$ 0.02 & 22.5 $\pm$ 0.01 & 4.9  $\pm$ 0.02 \\
%  4 & 1.5 $\pm$ 0.01 & 1.6  $\pm$ 0.03 & 19.7 $\pm$ 0.04 & 11.0 $\pm$ 0.04 & 7.7  $\pm$ 0.04 & 1.4  $\pm$ 0.04 \\
%  5 & 1.9 $\pm$ 0.08 & 12.9 $\pm$ 0.04 & 7.9  $\pm$ 0.06 & 22.8 $\pm$ 0.05 & 4.8  $\pm$ 0.05 & 30.3 $\pm$ 0.04 \\
%  6 & 1.8 $\pm$ 0.06 & 6.6  $\pm$ 0.03 & 22.6 $\pm$ 0.05 & 29.0 $\pm$ 0.06 & 13.3 $\pm$ 0.06 & 14.8 $\pm$ 0.05 \\
%  7 & 2.1 $\pm$ 0.08 & 2.3  $\pm$ 0.11 & 6.3  $\pm$ 0.14 & 19.1 $\pm$ 0.13 & 2.5  $\pm$ 0.13 & 16.3 $\pm$ 0.12 \\
%  8 & 1.5 $\pm$ 0.09 & 9.4  $\pm$ 0.07 & 12.2 $\pm$ 0.09 & 17.6 $\pm$ 0.12 & 26.7 $\pm$ 0.11 & 21.0 $\pm$ 0.09 \\
%  9 & 2.1 $\pm$ 0.04 & 5.5  $\pm$ 0.06 & 11.4 $\pm$ 0.04 & 30.2 $\pm$ 0.04 & 14.1 $\pm$ 0.04 & 25.9 $\pm$ 0.03 \\
% 10 & 1.9 $\pm$ 0.00 & 6.5  $\pm$ 0.00 & 16.0 $\pm$ 0.00 & 3.1  $\pm$ 0.00 & 12.9 $\pm$ 0.00 & 3.3  $\pm$ 0.00 \\
% 11 & 1.7 $\pm$ 0.05 & 8.8  $\pm$ 0.08 & 11.0 $\pm$ 0.05 & 2.5  $\pm$ 0.04 & 29.8 $\pm$ 0.04 & 24.2 $\pm$ 0.06 \\
% 12 & 1.7 $\pm$ 0.01 & 13.7 $\pm$ 0.01 & 27.9 $\pm$ 0.00 & 10.6 $\pm$ 0.00 & 12.2 $\pm$ 0.00 & 24.6 $\pm$ 0.01 \\
% 13 & 1.5 $\pm$ 0.11 & 4.5  $\pm$ 0.09 & 1.7  $\pm$ 0.15 & 1.9  $\pm$ 0.16 & 7.2  $\pm$ 0.18 & 24.3 $\pm$ 0.17 \\
% 14 & 1.6 $\pm$ 0.01 & 13.7 $\pm$ 0.07 & 18.7 $\pm$ 0.02 & 21.1 $\pm$ 0.03 & 24.0 $\pm$ 0.03 & 8.8  $\pm$ 0.03 \\
% 15 & 2.5 $\pm$ 0.01 & 14.5 $\pm$ 0.07 & 16.7 $\pm$ 0.02 & 11.8 $\pm$ 0.03 & 26.5 $\pm$ 0.03 & 15.3 $\pm$ 0.03 \\
% 16 & 1.1 $\pm$ 0.01 & 3.0  $\pm$ 0.07 & 7.7  $\pm$ 0.02 & 25.8 $\pm$ 0.03 & 23.6 $\pm$ 0.03 & 30.3 $\pm$ 0.03 \\\hline
% 17 & 2.6 $\pm$ 0.00 & 8.1  $\pm$ 0.00 & 5.4  $\pm$ 0.00 & 19.0 $\pm$ 0.00 & 15.6 $\pm$ 0.00 & 23.9 $\pm$ 0.00 \\
% 18 & 1.6 $\pm$ 0.05 & 12.5 $\pm$ 0.08 & 1.2  $\pm$ 0.05 & 15.0 $\pm$ 0.04 & 21.8 $\pm$ 0.04 & 10.0 $\pm$ 0.06 \\
% 19 & 2.4 $\pm$ 0.01 & 1.2  $\pm$ 0.01 & 17.9 $\pm$ 0.00 & 28.4 $\pm$ 0.00 & 4.3  $\pm$ 0.00 & 18.5 $\pm$ 0.01 \\
% 20 & 2.1 $\pm$ 0.11 & 12.7 $\pm$ 0.09 & 16.4 $\pm$ 0.15 & 28.5 $\pm$ 0.16 & 15.3 $\pm$ 0.18 & 29.9 $\pm$ 0.17 \\
% 21 & 1.3 $\pm$ 0.01 & 15.0 $\pm$ 0.07 & 11.0 $\pm$ 0.02 & 7.6  $\pm$ 0.03 & 21.0 $\pm$ 0.03 & 27.8 $\pm$ 0.03 \\
% 22 & 1.8 $\pm$ 0.01 & 4.4  $\pm$ 0.07 & 11.8 $\pm$ 0.02 & 1.3  $\pm$ 0.03 & 11.1 $\pm$ 0.03 & 26.4 $\pm$ 0.03 \\
% 23 & 1.9 $\pm$ 0.01 & 13.5 $\pm$ 0.07 & 4.3  $\pm$ 0.02 & 18.4 $\pm$ 0.03 & 11.4 $\pm$ 0.03 & 26.0 $\pm$ 0.03 \\\hline
%Avg & {\bf  1.87 $\pm$ 0.05}
%    & {\bf  8.86 $\pm$ 0.05}
%    & {\bf 12.70 $\pm$ 0.05}
%    & {\bf 16.96 $\pm$ 0.05}
%    & {\bf 14.80 $\pm$ 0.05}
%    & {\bf 18.11 $\pm$ 0.05} \\
%\hline
%\end{tabular}
%}
%%\end{small}
%\end{center}
%%\vspace*{-1cm}
%\end{table}
%\setlength{\tabcolsep}{1.4pt}
%
%%-------------------------------------------------------------------------------------------------------------
%%\begin{onecolumn}
%%begin{table}[!ht]
%\begin{table}[!h]
%\begin{center}
%\caption{Accuracy of  rulesets with standard deviation ({\em Acc
%  $\pm$ sd}) for different variants of the ordered algorithm using
%  10-fold stratified cross-validation.}
%\label{tab-results-Accuracy-Ordered}
%\vspace*{0.3cm}
%{\footnotesize
%%\begin{small}
%\begin{tabular}{crrrrrr}
%\hline\noalign{\smallskip}
%%\tiny
%          & \multicolumn{1}{c}{\bf CN2}
%          & \multicolumn{1}{c}{\bf CN2}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD} \\
%    \#    & \multicolumn{1}{c}{\bf standard}
%          & \multicolumn{1}{c}{\bf WRAcc}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
%          & \multicolumn{1}{c}{\bf (add.)} \\
%          & \multicolumn{1}{c}{\em ACC $\pm$ sd} &
%            \multicolumn{1}{c}{\em ACC $\pm$ sd} &
%            \multicolumn{1}{c}{\em ACC $\pm$ sd} &
%            \multicolumn{1}{c}{\em ACC $\pm$ sd} &
%            \multicolumn{1}{c}{\em ACC $\pm$ sd} &
%            \multicolumn{1}{c}{\em ACC $\pm$ sd} \\
%
%\noalign{\smallskip}
%\hline
%\noalign{\smallskip}
%  1 & 81.89 $\pm$ 0.01 & 85.38 $\pm$ 0.00 & 86.88  $\pm$ 0.03 & 87.01  $\pm$ 0.03 & 87.96  $\pm$ 0.03 & 87.04  $\pm$ 0.03 \\
%  2 & 93.84 $\pm$ 0.10 & 94.71 $\pm$ 0.04 & 96.65  $\pm$ 0.05 & 96.90  $\pm$ 0.04 & 98.06  $\pm$ 0.05 & 97.30  $\pm$ 0.04 \\
%  3 & 85.36 $\pm$ 0.03 & 84.45 $\pm$ 0.03 & 85.81  $\pm$ 0.02 & 86.80  $\pm$ 0.02 & 88.03  $\pm$ 0.01 & 88.07  $\pm$ 0.02 \\
%  4 & 98.07 $\pm$ 0.01 & 94.28 $\pm$ 0.03 & 95.61  $\pm$ 0.04 & 96.39  $\pm$ 0.04 & 96.66  $\pm$ 0.04 & 97.88  $\pm$ 0.04 \\
%  5 & 70.42 $\pm$ 0.08 & 73.16 $\pm$ 0.04 & 74.86  $\pm$ 0.06 & 75.56  $\pm$ 0.05 & 77.50  $\pm$ 0.05 & 75.74  $\pm$ 0.04 \\
%  6 & 67.17 $\pm$ 0.06 & 67.11 $\pm$ 0.03 & 69.04  $\pm$ 0.05 & 69.85  $\pm$ 0.06 & 71.18  $\pm$ 0.06 & 70.85  $\pm$ 0.05 \\
%  7 & 72.00 $\pm$ 0.08 & 67.90 $\pm$ 0.11 & 68.00  $\pm$ 0.14 & 68.23  $\pm$ 0.13 & 69.38  $\pm$ 0.13 & 68.64  $\pm$ 0.12 \\
%  8 & 68.92 $\pm$ 0.09 & 77.43 $\pm$ 0.07 & 79.06  $\pm$ 0.09 & 79.43  $\pm$ 0.12 & 79.44  $\pm$ 0.11 & 80.45  $\pm$ 0.09 \\
%  9 & 81.25 $\pm$ 0.04 & 80.00 $\pm$ 0.06 & 81.75  $\pm$ 0.04 & 82.49  $\pm$ 0.04 & 84.25  $\pm$ 0.04 & 84.01  $\pm$ 0.03 \\
% 10 & 98.64 $\pm$ 0.00 & 97.82 $\pm$ 0.00 & 99.48  $\pm$ 0.00 & 99.48  $\pm$ 0.00 & 99.80  $\pm$ 0.00 & 99.15  $\pm$ 0.00 \\
% 11 & 86.28 $\pm$ 0.05 & 87.72 $\pm$ 0.08 & 88.03  $\pm$ 0.05 & 88.61  $\pm$ 0.04 & 90.39  $\pm$ 0.04 & 88.87  $\pm$ 0.06 \\
% 12 & 95.32 $\pm$ 0.01 & 93.33 $\pm$ 0.01 & 94.29  $\pm$ 0.00 & 95.16  $\pm$ 0.00 & 95.20  $\pm$ 0.00 & 96.13  $\pm$ 0.01 \\
% 13 & 76.05 $\pm$ 0.11 & 77.56 $\pm$ 0.09 & 79.32  $\pm$ 0.15 & 80.14  $\pm$ 0.16 & 80.38  $\pm$ 0.18 & 81.65  $\pm$ 0.17 \\
% 14 & 73.87 $\pm$ 0.01 & 78.18 $\pm$ 0.07 & 78.38  $\pm$ 0.02 & 78.53  $\pm$ 0.03 & 79.34  $\pm$ 0.03 & 79.89  $\pm$ 0.03 \\
% 15 & 85.72 $\pm$ 0.01 & 76.38 $\pm$ 0.07 & 76.87  $\pm$ 0.02 & 77.26  $\pm$ 0.03 & 78.84  $\pm$ 0.03 & 78.20  $\pm$ 0.03 \\
% 16 & 93.33 $\pm$ 0.01 & 94.48 $\pm$ 0.07 & 95.78  $\pm$ 0.02 & 96.00  $\pm$ 0.03 & 97.46  $\pm$ 0.03 & 97.39  $\pm$ 0.03 \\\hline
% 17 & 79.19 $\pm$ 0.00 & 73.77 $\pm$ 0.00 & 74.71  $\pm$ 0.00 & 74.92  $\pm$ 0.00 & 75.74  $\pm$ 0.00 & 76.02  $\pm$ 0.00 \\
% 18 & 92.65 $\pm$ 0.05 & 81.37 $\pm$ 0.08 & 82.55  $\pm$ 0.05 & 82.75  $\pm$ 0.04 & 84.70  $\pm$ 0.04 & 83.53  $\pm$ 0.06 \\
% 19 & 64.44 $\pm$ 0.01 & 64.42 $\pm$ 0.01 & 65.94  $\pm$ 0.00 & 66.26  $\pm$ 0.00 & 68.08  $\pm$ 0.00 & 66.32  $\pm$ 0.01 \\
% 20 & 86.40 $\pm$ 0.11 & 17.74 $\pm$ 0.09 & 18.61  $\pm$ 0.15 & 18.97  $\pm$ 0.16 & 20.63  $\pm$ 0.18 & 20.03  $\pm$ 0.17 \\
% 21 & 88.89 $\pm$ 0.01 & 67.49 $\pm$ 0.07 & 67.92  $\pm$ 0.02 & 68.55  $\pm$ 0.03 & 68.74  $\pm$ 0.03 & 70.20  $\pm$ 0.03 \\
% 22 & 77.90 $\pm$ 0.01 & 76.36 $\pm$ 0.07 & 76.39  $\pm$ 0.02 & 77.25  $\pm$ 0.03 & 78.68  $\pm$ 0.03 & 77.50  $\pm$ 0.03 \\
% 23 & 91.49 $\pm$ 0.01 & 90.87 $\pm$ 0.07 & 92.68  $\pm$ 0.02 & 93.07  $\pm$ 0.03 & 94.86  $\pm$ 0.03 & 93.29  $\pm$ 0.03 \\\hline
%Avg & {\bf 83.00 $\pm$ 0.05}
%    & {\bf 78.34 $\pm$ 0.05}
%    & {\bf 79.50 $\pm$ 0.05}
%    & {\bf 79.98 $\pm$ 0.05}
%    & {\bf 81.14 $\pm$ 0.05}
%    & {\bf 80.83 $\pm$ 0.05} \\
%\hline
%\end{tabular}
%}
%%\end{small}
%\end{center}
%%\vspace*{-1cm}
%\end{table}
%\setlength{\tabcolsep}{1.4pt}
%
%
%\newpage
%
%\appendix
%\section*{Appendix C.}
%\subsection*{Tables with detailed results for different variants of the ordered algorithm in traffic accident data sets}
%\label{app-traffic}
%
%
%%-------------------------------------------------------------------------------------------------------------
%%\begin{onecolumn}
%%begin{table}[!ht]
%\begin{table}[!h]
%\begin{center}
%\caption{Average coverage per rule with standard deviation
%  ({\em COV $\pm$ sd}) for different variants of the ordered
%  algorithm using 10-fold stratified cross-validation.}
%\label{tab-traffic-Coverage-Ordered}
%\vspace*{0.3cm}
%{\footnotesize
%%\begin{small}
%\begin{tabular}{crrrrrr}
%\hline\noalign{\smallskip}
%%\tiny
%          & \multicolumn{1}{c}{\bf CN2}
%          & \multicolumn{1}{c}{\bf CN2}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD} \\
%    \#    & \multicolumn{1}{c}{\bf standard}
%          & \multicolumn{1}{c}{\bf WRAcc}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
%          & \multicolumn{1}{c}{\bf (add.)} \\
%          & \multicolumn{1}{c}{\em COV $\pm$ sd} &
%            \multicolumn{1}{c}{\em COV $\pm$ sd} &
%            \multicolumn{1}{c}{\em COV $\pm$ sd} &
%            \multicolumn{1}{c}{\em COV $\pm$ sd} &
%            \multicolumn{1}{c}{\em COV $\pm$ sd} &
%            \multicolumn{1}{c}{\em COV $\pm$ sd} \\
%
%\noalign{\smallskip}
%\hline
%\noalign{\smallskip}
%  1 & 108.5 $\pm$ 0.01 & 415.1 $\pm$ 0.00 & 451.0 $\pm$ 0.03 & 424.5 $\pm$ 0.03 & 436.8 $\pm$ 0.03 & 560.4 $\pm$ 0.03 \\
%  2 & 125.4 $\pm$ 0.10 & 153.3 $\pm$ 0.04 & 186.9 $\pm$ 0.05 & 218.0 $\pm$ 0.04 & 213.0 $\pm$ 0.05 & 161.3 $\pm$ 0.04 \\
%  3 & 256.4 $\pm$ 0.03 & 837.7 $\pm$ 0.03 & 881.2 $\pm$ 0.02 & 838.9 $\pm$ 0.02 & 932.4 $\pm$ 0.01 & 853.3 $\pm$ 0.02 \\
%  4 & 132.8 $\pm$ 0.01 & 250.5 $\pm$ 0.03 & 295.7 $\pm$ 0.04 & 251.6 $\pm$ 0.04 & 274.6 $\pm$ 0.04 & 311.2 $\pm$ 0.04 \\
%  5 & 159.7 $\pm$ 0.08 & 214.2 $\pm$ 0.04 & 260.9 $\pm$ 0.06 & 303.8 $\pm$ 0.05 & 281.2 $\pm$ 0.05 & 253.1 $\pm$ 0.04 \\
%  6 & 195.3 $\pm$ 0.06 & 534.7 $\pm$ 0.03 & 539.5 $\pm$ 0.05 & 615.4 $\pm$ 0.06 & 619.3 $\pm$ 0.06 & 595.8 $\pm$ 0.05 \\
%  7 & 439.1 $\pm$ 0.08 & 575.8 $\pm$ 0.11 & 596.7 $\pm$ 0.14 & 645.9 $\pm$ 0.13 & 576.8 $\pm$ 0.13 & 736.1 $\pm$ 0.12 \\\hline
%Avg & {\bf 202.44 $\pm$ 0.05}
%    & {\bf 425.91 $\pm$ 0.05}
%    & {\bf 458.84 $\pm$ 0.05}
%    & {\bf 471.17 $\pm$ 0.05}
%    & {\bf 476.33 $\pm$ 0.05}
%    & {\bf 495.88 $\pm$ 0.05} \\
%\hline
%\end{tabular}
%}
%%\end{small}
%\end{center}
%%\vspace*{-1cm}
%\end{table}
%\setlength{\tabcolsep}{1.4pt}
%
%
%%-------------------------------------------------------------------------------------------------------------
%%\begin{onecolumn}
%%begin{table}[!ht]
%\begin{table}[!h]
%\begin{center}
%\caption{Sizes of rulesets with standard deviation ({\em S
%  $\pm$ sd}) for different variants of the ordered algorithm using
%  10-fold stratified cross-validation.}
%\label{tab-traffic-Size-Ordered}
%\vspace*{0.3cm}
%{\footnotesize
%%\begin{small}
%\begin{tabular}{crrrrrr}
%\hline\noalign{\smallskip}
%%\tiny
%          & \multicolumn{1}{c}{\bf CN2}
%          & \multicolumn{1}{c}{\bf CN2}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD} \\
%    \#    & \multicolumn{1}{c}{\bf standard}
%          & \multicolumn{1}{c}{\bf WRAcc}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
%          & \multicolumn{1}{c}{\bf (add.)} \\
%          & \multicolumn{1}{c}{\em S $\pm$ sd} &
%            \multicolumn{1}{c}{\em S $\pm$ sd} &
%            \multicolumn{1}{c}{\em S $\pm$ sd} &
%            \multicolumn{1}{c}{\em S $\pm$ sd} &
%            \multicolumn{1}{c}{\em S $\pm$ sd} &
%            \multicolumn{1}{c}{\em S $\pm$ sd} \\
%
%\noalign{\smallskip}
%\hline
%\noalign{\smallskip}
%  1 & 21.7 $\pm$ 0.01 & 6.3  $\pm$ 0.00 & 9.2  $\pm$ 0.03 & 12.8 $\pm$ 0.03 & 14.0 $\pm$ 0.03 & 9.5  $\pm$ 0.03 \\
%  2 & 18.9 $\pm$ 0.10 & 16.5 $\pm$ 0.04 & 10.0 $\pm$ 0.05 & 13.1 $\pm$ 0.04 & 16.2 $\pm$ 0.05 & 13.5 $\pm$ 0.04 \\
%  3 & 9.7  $\pm$ 0.03 & 3.0  $\pm$ 0.03 & 6.3  $\pm$ 0.02 & 7.3  $\pm$ 0.02 & 8.3  $\pm$ 0.01 & 6.5  $\pm$ 0.02 \\
%  4 & 18.7 $\pm$ 0.01 & 10.0 $\pm$ 0.03 & 9.9  $\pm$ 0.04 & 12.3 $\pm$ 0.04 & 16.3 $\pm$ 0.04 & 10.8 $\pm$ 0.04 \\
%  5 & 15.7 $\pm$ 0.08 & 11.8 $\pm$ 0.04 & 11.2 $\pm$ 0.06 & 14.4 $\pm$ 0.05 & 15.1 $\pm$ 0.05 & 11.8 $\pm$ 0.04 \\
%  6 & 12.8 $\pm$ 0.06 & 4.8  $\pm$ 0.03 & 7.7  $\pm$ 0.05 & 10.7 $\pm$ 0.06 & 10.9 $\pm$ 0.06 & 10.0 $\pm$ 0.05 \\
%  7 & 6.3  $\pm$ 0.08 & 4.9  $\pm$ 0.11 & 6.5  $\pm$ 0.14 & 8.2  $\pm$ 0.13 & 8.6  $\pm$ 0.13 & 7.3  $\pm$ 0.12 \\\hline
%Avg & {\bf 14.83 $\pm$ 0.05}
%    & {\bf  8.19 $\pm$ 0.05}
%    & {\bf  8.69 $\pm$ 0.05}
%    & {\bf 11.26 $\pm$ 0.05}
%    & {\bf 12.77 $\pm$ 0.05}
%    & {\bf  9.91 $\pm$ 0.05} \\
%\hline
%\end{tabular}
%}
%%\end{small}
%\end{center}
%%\vspace*{-1cm}
%\end{table}
%\setlength{\tabcolsep}{1.4pt}
%
%
%
%%-------------------------------------------------------------------------------------------------------------
%%\begin{onecolumn}
%%begin{table}[!ht]
%\begin{table}[!h]
%\begin{center}
%\caption{Average significance of a rule with standard deviation ({\em SIG
%  $\pm$ sd}) for different variants of the ordered algorithm using
%  10-fold stratified cross-validation.}
%\label{tab-traffic-Significance-Ordered}
%\vspace*{0.3cm}
%{\footnotesize
%%\begin{small}
%\begin{tabular}{crrrrrr}
%\hline\noalign{\smallskip}
%%\tiny
%          & \multicolumn{1}{c}{\bf CN2}
%          & \multicolumn{1}{c}{\bf CN2}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD} \\
%    \#    & \multicolumn{1}{c}{\bf standard}
%          & \multicolumn{1}{c}{\bf WRAcc}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
%          & \multicolumn{1}{c}{\bf (add.)} \\
%          & \multicolumn{1}{c}{\em SIG $\pm$ sd} &
%            \multicolumn{1}{c}{\em SIG $\pm$ sd} &
%            \multicolumn{1}{c}{\em SIG $\pm$ sd} &
%            \multicolumn{1}{c}{\em SIG $\pm$ sd} &
%            \multicolumn{1}{c}{\em SIG $\pm$ sd} &
%            \multicolumn{1}{c}{\em SIG $\pm$ sd} \\
%
%\noalign{\smallskip}
%\hline
%\noalign{\smallskip}
%  1 & 1.6 $\pm$ 0.01 & 6.9 $\pm$ 0.00 & 8.1  $\pm$ 0.03 & 9.5  $\pm$ 0.03 & 9.0  $\pm$ 0.03 & 9.7  $\pm$ 0.03 \\
%   2 & 1.6 $\pm$ 0.10 & 6.4 $\pm$ 0.04 & 9.9  $\pm$ 0.05 & 9.3  $\pm$ 0.04 & 9.4  $\pm$ 0.05 & 9.4  $\pm$ 0.04 \\
%   3 & 1.0 $\pm$ 0.03 & 6.6 $\pm$ 0.03 & 8.9  $\pm$ 0.02 & 10.0 $\pm$ 0.02 & 11.0 $\pm$ 0.01 & 9.8  $\pm$ 0.02 \\
%   4 & 1.9 $\pm$ 0.01 & 7.5 $\pm$ 0.03 & 8.1  $\pm$ 0.04 & 10.9 $\pm$ 0.04 & 9.6  $\pm$ 0.04 & 11.0 $\pm$ 0.04 \\
%   5 & 1.3 $\pm$ 0.08 & 6.1 $\pm$ 0.04 & 10.5 $\pm$ 0.06 & 9.9  $\pm$ 0.05 & 12.0 $\pm$ 0.05 & 9.0  $\pm$ 0.04 \\
%   6 & 1.1 $\pm$ 0.06 & 8.5 $\pm$ 0.03 & 8.8  $\pm$ 0.05 & 9.9  $\pm$ 0.06 & 9.9  $\pm$ 0.06 & 11.5 $\pm$ 0.05 \\
%   7 & 1.5 $\pm$ 0.08 & 6.6 $\pm$ 0.11 & 8.5  $\pm$ 0.14 & 9.2  $\pm$ 0.13 & 9.7  $\pm$ 0.13 & 10.0 $\pm$ 0.12 \\\hline
%Avg & {\bf  1.43 $\pm$ 0.05}
%    & {\bf  6.93 $\pm$ 0.05}
%    & {\bf  8.97 $\pm$ 0.05}
%    & {\bf  9.81 $\pm$ 0.05}
%    & {\bf 10.09 $\pm$ 0.05}
%    & {\bf 10.05 $\pm$ 0.05} \\
%\hline
%\end{tabular}
%}
%%\end{small}
%\end{center}
%%\vspace*{-1cm}
%\end{table}
%\setlength{\tabcolsep}{1.4pt}
%
%%-------------------------------------------------------------------------------------------------------------
%%\begin{onecolumn}
%%begin{table}[!ht]
%\begin{table}[!h]
%\begin{center}
%\caption{Accuracy of rulesets with standard deviation ({\em Acc
%  $\pm$ sd}) for different variants of the ordered algorithm using
%  10-fold stratified cross-validation.}
%\label{tab-traffic-Accuracy-Ordered}
%\vspace*{0.3cm}
%{\footnotesize
%%\begin{small}
%\begin{tabular}{crrrrrr}
%\hline\noalign{\smallskip}
%%\tiny
%          & \multicolumn{1}{c}{\bf CN2}
%          & \multicolumn{1}{c}{\bf CN2}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD}
%          & \multicolumn{1}{c}{\bf CN2-SD} \\
%    \#    & \multicolumn{1}{c}{\bf standard}
%          & \multicolumn{1}{c}{\bf WRAcc}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.5$)}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.7$)}
%          & \multicolumn{1}{c}{\bf ($\gamma = 0.9$)}
%          & \multicolumn{1}{c}{\bf (add.)} \\
%          & \multicolumn{1}{c}{\em ACC $\pm$ sd} &
%            \multicolumn{1}{c}{\em ACC $\pm$ sd} &
%            \multicolumn{1}{c}{\em ACC $\pm$ sd} &
%            \multicolumn{1}{c}{\em ACC $\pm$ sd} &
%            \multicolumn{1}{c}{\em ACC $\pm$ sd} &
%            \multicolumn{1}{c}{\em ACC $\pm$ sd} \\
%
%\noalign{\smallskip}
%\hline
%\noalign{\smallskip}
%  1 & 73.48 $\pm$ 0.01 & 73.65 $\pm$ 0.00 & 74.12 $\pm$ 0.03 & 74.20 $\pm$ 0.03 & 73.73 $\pm$ 0.03 & 74.12 $\pm$ 0.03 \\
%  2 & 67.32 $\pm$ 0.10 & 64.88 $\pm$ 0.04 & 67.88 $\pm$ 0.05 & 68.15 $\pm$ 0.04 & 67.55 $\pm$ 0.05 & 67.98 $\pm$ 0.04 \\
%  3 & 87.09 $\pm$ 0.03 & 86.73 $\pm$ 0.03 & 87.09 $\pm$ 0.02 & 87.09 $\pm$ 0.02 & 86.61 $\pm$ 0.01 & 87.09 $\pm$ 0.02 \\
%  4 & 70.71 $\pm$ 0.01 & 69.39 $\pm$ 0.03 & 70.27 $\pm$ 0.04 & 70.27 $\pm$ 0.04 & 69.59 $\pm$ 0.04 & 70.35 $\pm$ 0.04 \\
%  5 & 74.78 $\pm$ 0.08 & 73.70 $\pm$ 0.04 & 74.66 $\pm$ 0.06 & 74.58 $\pm$ 0.05 & 73.98 $\pm$ 0.05 & 74.58 $\pm$ 0.04 \\
%  6 & 84.88 $\pm$ 0.06 & 84.76 $\pm$ 0.03 & 84.88 $\pm$ 0.05 & 84.88 $\pm$ 0.06 & 84.69 $\pm$ 0.06 & 84.76 $\pm$ 0.05 \\
%  7 & 82.57 $\pm$ 0.08 & 82.49 $\pm$ 0.11 & 82.70 $\pm$ 0.14 & 82.70 $\pm$ 0.13 & 82.57 $\pm$ 0.13 & 82.67 $\pm$ 0.12 \\\hline
%Avg & {\bf 77.26 $\pm$ 0.05}
%    & {\bf 76.51 $\pm$ 0.05}
%    & {\bf 77.37 $\pm$ 0.05}
%    & {\bf 77.41 $\pm$ 0.05}
%    & {\bf 76.96 $\pm$ 0.05}
%    & {\bf 77.36 $\pm$ 0.05} \\
%\hline
%\end{tabular}
%}
%%\end{small}
%\end{center}
%%\vspace*{-1cm}
%\end{table}
%\setlength{\tabcolsep}{1.4pt}
%
%
%\end{onecolumn}
%
%\end{document}
