Category Archives: Uncategorized

Oracle Enterprise Linux for Home User: Install Latest Firefox Version

It is a joy to use Linux for any purpose. Although Oracle Linux is mainly designed for enterprise stack, a stable operating system for a technical user is always a need.

However default configurations and program versions are not suitable for daily use always. One example of this is Firefox 3.0.6 available in Oracle Linux (or Red Hat). Here is how you can upgrade it to 17.0

Download latest Firefox release from Mozilla web page. Then unpack and install it by simply following.

tar -xjvf firefox-17.0.tar.bz2
cp -r firefox /opt/
ln -sf /opt/firefox/firefox /usr/bin/firefox

Line of Sight (LoS) Analysis: Optimizing the Observers for Best Coverage (Part 4)

After my last post on multiple observers, I will discuss another topic related to multiple observers. The question is “What should be the optimal choice of N observers on terrain such that visible region (as many green points as possible by our convention) is maximized ?”.

We first define a pseudo code in order to find the optimal (Not guaranteed. Keep in mind that optimization problems are usually NP-complete by their nature) layout of N observers. For simplicity we will assume that all observers have the same height (7 units) which can be relaxed later.

We will implement a constructive way of finding optimal layout for N observers. Here is the pseudo code:

1. Find the optimal layout for 1 observer and compute coverage ratio (best coverage for one observer)
2. Add another random observer ((uniform(-8,8),uniform(-8,8))) and compute the coverage for those two observers (random one and the best observer from Step 1).
1. If the new coverage is better than the coverage in Step 1, use this as the input of optimization solver
2. Otherwise repeat Step 2 to find a better coverage.
3. For number of observers greater than 2 apply the idea in Step 2 recursively.

There are some blur points in this pseudo code. We will define those before moving further with the implementation.

Coverage

The very first thing to be defined is the coverage idea. As you will remember from second post, we have defined our 3D terrain by evaluating our height function over outer product of x & y values varying over [-8,8] with a step size of 0.1 units. We have 1681 different (x,y) tuples. Here is the definition of coverage based on our conventions:

• Coverage Ratio is the ratio of points within LoS of a given observer/group of observers (at least one of the observers mark those set of points as green) to the total number of points (1681)

How to Find Optimal Coordinates of Observers ?

Optimality is a very common word used in place of many different concepts  in real life or engineering. Let me define it once more for our purpose:

• Optimization is the process of searching for an N-dimensional vector using a technique to maximize/minimize a function of that N-dimensional vector.

Now let’s substitute three italic words of definition for our problem:

• N-dimensional vector in our problem is the vector of first to components of observer dimensions. Such as, (x1,y1,x2,y2,…,xn,yn).
• Technique to be used is the Nelder and Mead Technique (A version of it implemented in R).
• Function to be maximized is the coverage function which we have defined for a given set of observers.

Implementation

Let’s start by defining the function to be optimized that is coverage of terrain for a given set of observers.

targetfunc<-function(observer){
m <- matrix(data=observer,ncol=2,byrow=TRUE)

# Compute merged status of all observers
mergedstatus <- rep("red",length(terrain$height)) for(oidx in seq(1:dim(m))){ terrain$dist2observer <- distance(terrain, c(m[oidx,],7))

status <- LoS(terrain,c(m[oidx,],7),maxVisibleDistance)

}

sum(mergedstatus=="green")/1681
}

matrix routine allows us to create a table of two columns(first two dimensions of observers) and length(observer)/2 rows. We have used the technique discussed in part 3 to compute merged status of observers.  sum(mergedstatus==”green”) call is used to count number of green points on terrain with respect to observers.

Next is the computation of first input to be given to optimization solver. That’s because for any optimization technique starting point is critical. Without any formal definition we will use our pseudo code to choose a “good starting point/vector”.

n <- 2
baselineValue <- 0.541344
previousObserver <- c(1.15861411217711, 1.1499851362913)
observers <- c(previousObserver,runif(2,-8,8))
while(targetfunc(observers) <= baselineValue){
observers <- c(previousObserver,runif(2,-8,8))
}
print(observers)

Above code is an example to initialize observers vector for searching best 2 observer layout. It uses the best coverage ratio for 1 observer case (54.1344%) and adds a new random observer next to best observer found for single observer case.

Final point is the optimization solver which is very simple and totally handled by R

optim <- optim(observers, targetfunc,
control=list(fnscale=-1,trace=5,REPORT=1))

First parameter is the initial value for input vector (prepared by previous code piece). Second parameter is the name of the function to be maximized. optim function is implemented to solve minimization problems by default. Setting fnscale attribute of control parameter turns it to a maximization problem solver.

Now we can combine all to have our final script

library(rgl)
##################
# Functions
##################
# 3D Terrain Function
height <- function (point) {
sin(point$x)+0.125*point$y*sin(2*point$x)+sin(point$y)+0.125*point$x*sin(2*point$y)+3
}

# Linear Function
linear <- function (px, observer, target) {
v <- observer - target

y <- ((px - observer)/v)*v+observer
z <- ((px - observer)/v)*v+observer

data.frame(x=px,y=y, z=z)
}

# Linear Function
distance <- function (terrain, observer) {
sqrt((terrain$x-observer)^2+(terrain$y-observer)^2+(terrain$height-observer)^2) } LoS <- function(terrain, observer, maxVisibleDistance){ status = c() for (i in seq(1:nrow(terrain))) { if (observer == terrain$x[i] && observer == terrain$y[i]){ if(observer >= terrain$height[i]){
if (terrain$dist2observer[i] > maxVisibleDistance){ status <- c(status,"yellow") }else{ status <- c(status,"green") } }else{ status <- c(status,"red") } }else{ # All points on line line <- linear(seq(from=min(observer,terrain$x[i]),
to=max(observer,terrain$x[i]), by=0.1), observer, c(terrain$x[i],terrain$y[i],terrain$height[i]))
# Terrain Height
h <- height(line)

# LoS Analysis
aboveTerrain <- round((line$z-h),2) >= 0.00 visible <- !is.element(FALSE,aboveTerrain) if (visible){ # Second Rule if(terrain$dist2observer[i] <= maxVisibleDistance){
status <- c(status,"green")
}else{
status <- c(status,"yellow")
}
}else{
status <- c(status,"red")
}
}
}

status
}

mergedstatus<-c()

for(i in seq(length(status1))){
if (status1[i] == "green" || status2[i] == "green"){
mergedstatus <- c(mergedstatus,"green")
}else if (status1[i] == "yellow" || status2[i] == "yellow"){
mergedstatus <- c(mergedstatus,"yellow")
}
else{
mergedstatus <- c(mergedstatus,"red")
}
}

mergedstatus

}

##################
# Input
##################

# Max visible distance
maxVisibleDistance = 8

# Generate points with a step size of 0.1
x <- seq(from=-8,to=8,by=0.4)

xygrid <- expand.grid(x=x,
y=x)

terrain <- data.frame(xygrid,
height=height(xygrid)
)

targetfunc<-function(observer){
#print(observer)
m <- matrix(data=observer,ncol=2,byrow=TRUE)

# Compute merged status of all observers
mergedstatus <- rep("red",length(terrain$height)) for(oidx in seq(1:dim(m))){ terrain$dist2observer <- distance(terrain, c(m[oidx,],7))

status <- LoS(terrain,c(m[oidx,],7),maxVisibleDistance)

}

sum(mergedstatus=="green")/1681
}

n <- 3
baselineValue <- 0.541344
previousObserver <- c(-1.32661956044593, 2.18870625357827)

# List of observers (x1,y1,z1,x2,y2,z2)
observers <- c(previousObserver,runif(2,-8,8))
while(targetfunc(observers) <= baselineValue){
observers <- c(previousObserver,runif(2,-8,8))
}
print(observers)

optim <- optim(observers, targetfunc,
control=list(fnscale=-1,trace=5,REPORT=1))

Results

Here is the coverage ratio for different number of observers after optimization You can test covering more than 98% of whole terrain is not trivial by using only 6 random observers but requires “careful” choice of their layout.

Finally let’s check step by step improvement in coverage as we add more optimal observers.

Single Observer Two Observers Three Observers Four Observers Five Observers Six Observers Line of Sight (LoS) Analysis: Multiple Observers (Part 3)

In this part of my LoS Analysis series, I will try to extend 3D LoS analysis for multiple observers. Assume that you drop multiple observers into a terrain with the aim of covering it perfectly (100% green).

We will reuse R codes used in Part 2. However we need to add a simple code piece to be used to merge Line of Sight results of multiple observers. If a point on terrain is visible by any of the observers that means point is visible, if the point is visible but far from all observers that means point is out of LoS due to distance (marked with yellow), for all other conditions point on terrain is red. updatestatus function is implemented for this purpose.

library(rgl)
##################
# Functions
##################
# 3D Terrain Function
height <- function (point) {
sin(point$x)+0.125*point$y*sin(2*point$x)+sin(point$y)+0.125*point$x*sin(2*point$y)+3
}

# Linear Function
linear <- function (px, observer, target) {
v <- observer - target

y <- ((px - observer)/v)*v+observer
z <- ((px - observer)/v)*v+observer

data.frame(x=px,y=y, z=z)
}

# Linear Function
distance <- function (terrain, observer) {
sqrt((terrain$x-observer)^2+(terrain$y-observer)^2+(terrain$height-observer)^2) } LoS <- function(terrain, observer, maxVisibleDistance){ status = c() for (i in seq(1:nrow(terrain))) { if (observer == terrain$x[i] && observer == terrain$y[i]){ if(observer >= terrain$height[i]){
if (terrain$dist2observer[i] > maxVisibleDistance){ status <- c(status,"yellow") }else{ status <- c(status,"green") } }else{ status <- c(status,"red") } }else{ # All points on line line <- linear(seq(from=min(observer,terrain$x[i]),
to=max(observer,terrain$x[i]), by=0.1), observer, c(terrain$x[i],terrain$y[i],terrain$height[i]))
# Terrain Height
h <- height(line)

# LoS Analysis
aboveTerrain <- round((line$z-h),2) >= 0.00 visible <- !is.element(FALSE,aboveTerrain) if (visible){ # Second Rule if(terrain$dist2observer[i] <= maxVisibleDistance){
status <- c(status,"green")
}else{
status <- c(status,"yellow")
}
}else{
status <- c(status,"red")
}
}
}

status
}

mergedstatus<-c()

for(i in seq(length(status1))){
if (status1[i] == "green" || status2[i] == "green"){
mergedstatus <- c(mergedstatus,"green")
}else if (status1[i] == "yellow" || status2[i] == "yellow"){
mergedstatus <- c(mergedstatus,"yellow")
}
else{
mergedstatus <- c(mergedstatus,"red")
}
}

mergedstatus

}

##################
# Input
##################
# Observer location
#observers<-c(0,0, 6,1,1,6)

# Max visible distance
maxVisibleDistance = 8

# Generate points with a step size of 0.1
x <- seq(from=-8,to=8,by=0.4)

xygrid <- expand.grid(x=x,
y=x)

terrain <- data.frame(xygrid,
height=height(xygrid)
)

# List of observers (x1,y1,z1,x2,y2,z2)
observers <- c(runif(2,-8,8),6,runif(2,-8,8),6,
runif(2,-8,8),6,runif(2,-8,8),6,
runif(2,-8,8),6,runif(2,-8,8),6,
runif(2,-8,8),6,runif(2,-8,8),6)
m <- matrix(data=observers,ncol=3,byrow=TRUE)

# Compute merged status of all observers
mergedstatus <- rep("red",length(terrain$height)) for(oidx in seq(1:dim(m))){ terrain$dist2observer <- distance(terrain, m[oidx,])

status <- LoS(terrain,m[oidx,],maxVisibleDistance)

}

# Set merged status as the ultimate status
terrain <- data.frame(terrain,status = mergedstatus)

rgl.open()
rgl.surface(x, x,
matrix(data=terrain$height,nrow=length(x),ncol=length(x)), col=matrix(data=mergedstatus,nrow=length(x),ncol=length(x)) ) bg3d("gray") # Mark all observers for(oidx in seq(1:dim(m))){ spheres3d(c(m[oidx,1]), c(m[oidx,3]), c(m[oidx,2]), radius=0.25, color="white" ) } rgl.viewpoint(-60,30) A Few Examples Here are a few examples. All those observers are uniformly distributed over terrain using runif function Trivial Case: Single Observer Two Observers Four Observers Eight Observers Line of Sight (LoS) Analysis: 3D Terrain Analysis (Part 2) In my previous post on LoS Analysis, I have tried to explain briefly the basics of LoS in two dimensional space. Obviously real life problems are based on three dimensional terrains although basic concepts are all the same. In this second part I will try to adapt the same techniques with a few modifications for three dimensional terrains. 3D Terrain Visualization with R One of the first differences in 3D LoS analysis is the terrain visualization. We can not use plot function for proper visualization is 3D. Fortunately R has all packages you need for any type of problem. I will use rgl package which can be downloaded using install.packages("rgl") command. Once you have the rgl package, generating pseudo 3D terrains as we did for 2D is a trivial thing. You can use the following R script to generate your 3D terrains like above. library(rgl) # 3D Terrain Function height <- function (x,y) { sin(x)+0.125*y*sin(2*x)+sin(y)+0.125*x*sin(2*y)+0.25 } # Terrain boundaries -8<=x<=8 and -8<=y<=8 boundary <- c(-8,8) # Terrain grid with a step size of 0.1 units xy<-seq(from=boundary,to=boundary,by=0.1) # Evaluate all heights for all grid points z<-outer(xy,xy,height) # A few visualization staff zlim <- range(z) zlen <- zlim - zlim + 1 colorlut <- terrain.colors(zlen) # height color lookup table col <- colorlut[ z-zlim+1 ] # assign colors to heights for each point # Draw the terrain rgl.open() bg3d("gray") rgl.surface(xy, xy, z, color=col) A new function in this script is outer function which generates the product of a vector and a row-vector to have a matrix (product of a row-vector with a vector/column-vector is obviously a scalar value and named to be dot/inner product). The third parameter of the function provides us the mechanism to apply a given function (height in our case) for each element of this matrix. Obviously you can play with height function to have fancier 3D terrains and to have best visualization you may need viewpoint routine in rgl package . LoS in 3D Terrain Line of Sight analysis on 3D terrain uses the same principles as it does in 2D. Use the following R script to decide on status of a point (invisible, visible, visible but far away) library(rgl) ################## # Functions ################## # 3D Terrain Function height <- function (x,y) { sin(x)+0.125*y*sin(2*x)+sin(y)+0.125*x*sin(2*y)+0.25 } # Linear Function linear <- function (x, observer, target) { v <- observer - target y <- ((x - observer)/v)*v+observer z <- ((x - observer)/v)*v+observer data.frame(x=x,y=y, z=z) } # Linear Function distance <- function (p0,p1) { sqrt(sum((p0-p1)^2)) } ################## # Input ################## # Observer location observer<-c(10,10,1) # Target on terrain target <- c(5, 5, height(5,5)) # Max visible distance maxVisibleDistance = 4 # Generate points with a step size of 0.1 x <- seq(from=min(observer,target), to=max(observer,target), by=0.1) # All points on line line <- linear(x, observer, target) # Terrain Height h <- height(line$x,line$y) # LoS Analysis aboveTerrain <- round((line$z-h),2) >= 0.1

# First Rule
visible <- !is.element(FALSE,aboveTerrain)
if (visible){
# Second Rule
d <- distance(observer, target)
if(d <= maxVisibleDistance){
status <- "LoS"
}else{
status <- "non-LoS due to Distance"
}
}else{
status <- "non-LoS due to Blocking"
}

Obviously there are a few changes in the script with compared to 2D version.  The first one is linear function(Code Lines 10-18). New version not only evaluates second (y) but also the third dimension (z). Notice that z is our height dimension by convention. We have also utilized data.frame function to concatenate all dimensions to form a table of point dimensions

The second difference is on height function (Code Lines 5-8). It is no longer a mapping from x to y but a mapping from x,y to z.

Rest of the 3D version of script is pretty much the same or trivial to discuss more.

Visualizing LoS on 3D Terrain

Until this point we have analyzed LoS of a single point on 2D-3D terrains. But usually network analists wish to know LoS map of the terrain with respect to a given observer.  In other words we need to visually understand which regions on 3D terrain are visible by the observer, invisible by the observer due to blocking, or further than the limit from the observer.

Here the LoS map of our pseudo 3D terrain with respect to an observer with a given set of coordinates and maximum service range(green vs yellow regions).

You can obtain this visualization using following R script.

library(rgl)
##################
# Functions
##################
# 3D Terrain Function
height <- function (point) {
sin(point$x)+0.125*point$y*sin(2*point$x)+sin(point$y)+0.125*point$x*sin(2*point$y)+3
}

# Linear Function
linear <- function (px, observer, target) {
v <- observer - target

y <- ((px - observer)/v)*v+observer
z <- ((px - observer)/v)*v+observer

data.frame(x=px,y=y, z=z)
}

# Linear Function
distance <- function (terrain, observer) {
sqrt((terrain$x-observer)^2+(terrain$y-observer)^2+(terrain$height-observer)^2) } LoS <- function(terrain, observer, maxVisibleDistance){ status = c() for (i in seq(1:nrow(terrain))) { if (observer == terrain$x[i] && observer == terrain$y[i]){ if(observer >= terrain$height[i]){
if (terrain$dist2observer[i] > maxVisibleDistance){ status <- c(status,"yellow") }else{ status <- c(status,"green") } }else{ status <- c(status,"red") } }else{ # All points on line line <- linear(seq(from=min(observer,terrain$x[i]),
to=max(observer,terrain$x[i]), by=0.1), observer, c(terrain$x[i],terrain$y[i],terrain$height[i]))
# Terrain Height
h <- height(line)

# LoS Analysis
aboveTerrain <- round((line$z-h),2) >= 0.00 visible <- !is.element(FALSE,aboveTerrain) if (visible){ # Second Rule if(terrain$dist2observer[i] <= maxVisibleDistance){
status <- c(status,"green")
}else{
status <- c(status,"yellow")
}
}else{
status <- c(status,"red")
}
}
}

status
}

##################
# Input
##################
# Observer location
observer<-c(0.835597146302462, -1.71025141328573, 6)

# Max visible distance
maxVisibleDistance = 8

# Generate points with a step size of 0.1
x <- seq(from=-8,to=8,by=0.4)

xygrid <- expand.grid(x=x,
y=x)

terrain <- data.frame(xygrid,
height=height(xygrid)
)

terrain <- data.frame(terrain,
dist2observer=distance(terrain, observer)
)

terrain <- data.frame(terrain,
status = LoS(terrain, observer, maxVisibleDistance))

rgl.open()
rgl.surface(x, x,
matrix(data=terrain$height,nrow=length(x),ncol=length(x)), col=matrix(data=terrain$status,nrow=length(x),ncol=length(x))
)
bg3d("gray")
# Mark the observer
spheres3d(c(observer),
c(observer),
c(observer),
color="white"
)

rgl.viewpoint(-60,30)

For a better visualization R allows you to implement spinning 3D terrains using play3d function and record it in gif format using movie3d function as I did below.

Introduction

Line of Sight analysis is a commonly used technique in telecommunication industry for A/I (Air Interface) equipment planning and allocation. With the simplest terms LoS is the question whether a point on N-dimensional space is visible by an other observer point. The question can be used to answer where to locate a transceiver on terrain so that it can serve customers on some region A.

Before relatively more complicated problems, let’s start with an easy example focusing on two dimensional terrains. Throughout the post, we will use R for coding which is my favorite option for any mathematical problem (statistics, plotting, linear algebra, optimization, etc.). But you can easily adapt coding material to Mathlab, Python,or your favorite language.

We will start by defining a mathematical function to be used to generate our pseudo terrains. For this purpose trigonometric functions (sin, cos) and polynomial functions are the best ones because of their wavy shapes. Here is an example of trigonometric terrain Figure 1 Trigonometric Terrain

In order to generate this two dimensional one use the following code piece

x <- seq(from=4,to=10,by=0.01)
y <- sin(x)+cos(2*x)+sin(3*x)+cos(4*x)+3

windows()
plot(x,y,'l',
main="y=sin(x)+cos(2x)+sin(3x)+cos(4x)+3",
ylab="height",col="blue")
You may choose to use a polynomial terrain also Figure 2 Polynomial Terrain

To obtain this terrain, use the following R script piece

x <- seq(from=0,to=6,by=0.01)
y <- x*(x-1)*(x-2)*(x-3)*(x-4)*(x-5)*(x-6)+100

windows()
plot(x,y,'l',
main="y=x(x-1)(x-2)(x-3)(x-4)(x-5)(x-6)+100",
ylab="height",col="blue")

Combining polynomial terrain functions with trigonometric ones will give you fancier ones.

What is LoS ?

You can think that we have already answered this question but this was an informal try which is not very useful for solving the problem. In order to solve this problem methodically we need to understand what makes a target visible (within LoS) by the observer.

As you see on Figure 3, green point is within line of sight of observer (blue point). However there is pseudo hill between red point and observer. The difference is that the line connecting observer and green point is always greater than the terrain function whereas this is not valid for the line connecting observer and red point (for x ε [~2.5, ~3.5] red line is under the terrain curve). Figure 3 LoS vs non-LoS

This was the first point (blocking) we should define. The second one is an easier one related with maximum Euclidean distance between observer and target. The distance between observer and target may cause a phase shift in signal if the distance is sufficiently long or depending on weather conditions and terrain properties you may observer diffraction problems (actually there might be more than those). In return this will cause signal quality issues or call drops.  On Figure 3, although blocking is not an issue between observer and  yellow point, target is out of  visible range (say 8 units) of observer.

You can generate Figure 3 using the following R script

# Terrain Function
height <- function (x) {
x*x/3+sin(x)+cos(2*x)+sin(3*x)+cos(4*x)+sin(5*x)+cos(6*x)+3
}

# Observer location
observer<-c(1.5,8.9)

# Generate terrain points with a tolerance of 0.1
x<-seq(from=-0.1,to=6.1,by=0.1)
terrainHeight<-height(x)

windows()
# Draw terrain
plot(x,terrainHeight,type='b',
xlim=range(x),ylim=range(terrainHeight),
main="Line of Sight (LoS)",
ylab="Height",xlab="")

# Not LoS
points(x=c(observer,x),
y=c(observer,terrainHeight),
col="red",type='b')

# LoS
points(x=c(observer,x),
y=c(observer,terrainHeight),
col="green",type='b')

# LoS but far
points(x=c(observer,x[length(x)]),
y=c(observer,terrainHeight[length(x)]),
col="yellow",type='b')

# Draw Observer
points(x=c(observer),
y=c(observer),
col="blue",pch=10)

Method to Decide LoS

Finally let’s define a method to find all visible, invisible, and “far” points on any terrain. Since it is not “easy” to decide analytically whether the line connecting observer and target “is above” the terrain for any terrain function, we will use a simple numeric method.

We will define a step size small enough (around Spatial Tolerance) to generate all x values between observer and target. seq function is a good choice for doing this (Code Lines 33-36).  Evaluate these x values for line function connecting observer and target and terrain function. Evaluation is simple for terrain function using height function (Code Lines 4-7). Evaluation of line function is held by function linear  using parametric definition of line function (Code Lines 9-14) . Next step is to search for any x value having a line evaluation less than terrain evaluation (Code Line 44-28). The rest is simple as to evaluate euclidean distance and assigning values to status variable.

##################
# Functions
##################
# Terrain Function
height <- function (x) {
x*x/3+sin(x)+cos(2*x)+sin(3*x)+cos(4*x)+sin(5*x)+cos(6*x)+3
}

# Linear Function
linear <- function (x, observer, target) {
v <- observer - target

((x - observer)/v)*v+observer
}

# Linear Function
distance <- function (p0,p1) {
sqrt(sum((p0-p1)^2))
}

##################
# Input
##################
# Observer location
observer<-c(1.5,9)

# Target on terrain
target <- c(5, height(5))

# Max visible distance
maxVisibleDistance = 4

# Generate points with a step size of 0.1
x <- seq(from=min(observer,target),
to=max(observer,target),
by=0.1)

# Terrain Height
h <- height(x)

# y Values
y <- linear(x, observer, target)

# LoS Analysis
aboveTerrain <- round((y-h),2) >= 0.00

# First Rule
visible <- !is.element(FALSE,aboveTerrain)
if (visible){
# Second Rule
d <- distance(observer, target)
if(d <= maxVisibleDistance){
status <- "LoS"
}else{
status <- "non-LoS due to Distance"
}
}else{
status <- "non-LoS due to Blocking"
}

My Comments on Voltaire InfiniBand (IB) As you all may know Voltaire is one of the most prominent IB technology vendors and investors had the partnership with Oracle in building Exadata Version 1. Yet another beautiful thing about Voltaire is that it is one of the companies invest on Open Source technologies and HPC (High Performance Computing) a lot, like OFED stack development on Linux.

You can find my comments about Voltaire 20Gbit IB solution which we have already used in our large (25TB) RAC consolidation project on Oracle 11g Release 1.

Business Worldwide Choose Voltaire InfiniBand to Enhance  Performance and Scalability of Oracle RAC

ORION IO Calibration Cookbook ORION is one of the most important calibration tools of my toolbox. In any project before installing any Oracle software (clusterware, ASM or RDBMS), the first thing I do is to calibrate disks of the system. That’s because history is full storage related Oracle stories.

In this post, you will find a simple cookbook of calibrating a storage array by using ORION. I will go through a real calibration test I did three months ago.

Preparing a .lun file

In any ORION test, the first thing you should do is to prepare a LUN file with extension .lun to list your devices (with the assumption that you will be testing raw devices. ORION also let you to test file systems in which you should use synthetically created large files in place of device names) to be calibrated. Here is our lun file

 [oracle@consol10g orion]$cat mytest.lun /dev/dm-2 /dev/dm-3 /dev/dm-4/dev/dm-5 /dev/dm-6 One important point is that you should ensure that the OS user you will be performing the test has the necessary grants over the devices you will be calibrating. Small Random & Large Sequential Read Load I always start with pure small (8K) random and then pure large (1M) sequential read load calibration first. That is simply because if these two tests fail to satisfy your performance needs there is no point in doing any other tests. You can simply call back your vendor or infrastructure team. Here is the ORION syntax we will be using  [oracle@consol10g orion]$ ./orion_lnx     -run advanced -testname mytest -num_disks 40 -simulate raid0 –write 0                                           –type seq –matrix basic –cache_size 67108864 -verbose

Let us discuss this parameterization. In order to control other options in detail I have set run option to be advanced. testname option should set to be the name of .lun file. num_disks is the number of disks in your storage array. Actually, you can set this parameter to any value you like it is just an input for calculating the maximum number of small IO and large IO requesters (for value less than 10 max small IO requesters= 5*num_disks and max large IO requesters = 2*num_disks).simulate parameter is always raid0 me, because I always use ASM. type option defines the type of large IOs (sequential/random). Since the test covers pure random and pure sequential IO loads, matrix option is set to be basic. cache_size is a critical parameter you should learn by get in touch with your storage admin. This is the size of your storage array cache in MB. If you set this parameter too low your outputs will be too good to be true, if you set too high your test run time will be too long. verbose is the parameter to guide ORION to print progress status on standard output.

Keep in mind that you should run ORION from the same directory where mytest.lun file resides. When the execution is over you will find three new files in your current working directory. Those are mytest_*_summary.txt, mytest_*_iops.txt, mytest_*_lat.txt, mytest_*_mbps.txt files. Figure 1 Pure Random and Pure Sequential Loads

As you see the number of read IOPS is increasing with the increasing number of outstanding random read requesters to a certain point. This is what we expect to see from queuing theory. However, there is another restrictive parameter over that, called latency. If the service time of a single read request exceeds 10ms on the average, (some others say 20ms) then the user will start to suffer.

So keep in mind that doing ~100.000 IOPS means nothing. What is meaningful is that you can deliver ~3500 IOPS with 19.5 ms service time. In other words never use IOPS metric barely. Always cascade average service time, latency, queuing time, etc metrics.

The interpretation of sequential IO is different. In high throughput demanding systems, nobody will deal with the service time of a single IO request. The import thing is to fully utilize the whole storage infrastructure in order to deliver the highest rate of throughput possible. Therefore, “160 MB/s read throughput” statement is OK.

The last thing you should test is the behavior of storage array under various mix loads. Let me exemplify the importance of this test. You might see some systems running in peace whole day long but 1-2 hours night time (backup window). Alternatively, you might see some DBAs periodically kill reporting users in OLTP systems.

Those are all related with insufficient IO configurations. Large IO requests caused by backup, reporting,etc activites may result in a severe change in the service time of small IO requests. Therefore, you should have a priori knowledge in what proportion, small and large IO requests can cause a problem on your storage infrastructure. Here is the ORION syntax

 [oracle@consol10g orion]\$ ./orion_lnx     -run advanced -testname mytest -num_disks 40 -simulate raid0 –write 0                                           –type seq –matrix detailed –cache_size 67108864 -verbose

Notice that the only change in here is that matrix option is set to be detailed. When matrix option is basic, ORION generates random IO at N different levels without sequential IO requesters and then it generates sequential IO at M different levels without random IO requesters. When the option equals to detailed, ORION generates all possible MxN combination of random & sequential IO genererators. To interpret Figure 4, let’s think that our storage array is capable of serving only 8K requests. Any larger requests will be chopped into 8K pieces. That means a large IO request will be corresponding to 125 small IO requests. Moreover think that the total capacity of our storage array is 2000 small IOPS. Now by simple division you can either yield 2000 small (8K) IOPS or 16 large (1M) IOPS from this storage array or somewhere between.

So when the number of total large IO requesters increase, the number of total IOPS will decrease.

Now assume that sustaining 1500 IOPS requires 10 ms, 3000 IOPS requires 20 ms service time on the average. While we are sustaining 1500 IOPS, we can either move on large requester axis and with an addition of 12 IOPS we can reach 20 ms latency, or we can move on small requester axis and with an addition of 1500 IOPS we can reach 20 ms latency (We may choose a third option somewhere between also). As a result increase in large IO results in an increase in service time also. Conclusion

In this post, I try to show how ORION can help you in detecting your possible disk performance problems before they occur on production. Remember that on an enterprise storage infrastructure there are more than just a bunch disks. The major performance problem can be related to components like HBAs, switches, port issues, backend problems, or even IO scheduling algorithm you use. Mathlab Codes to Create Charts

In order to generate the plots you’ve seen in this post you can use orion_chart_basic.m and orion_chart_detailed.m scripts with mathlab. When you execute them(F5), they will ask you to choose a directory. This directory should contain all ORION output files of a single run. Once you pick the directory, mathlab will do the rest.

CAUTIONS

• For tests executed with run option set to be basic you should use orion_chart_basic.m
• For tests executed with run parameter set to be detailed you should use orion_chart_detailed.m
• Each directory you choose should contain one and only one set of ORION run output file.
• All four mathlab files should be in the same working directory.

randomIOChart.m

 function randomIOchart(concSmallIO, IOPS, latency) [haxes,hline1,hline2] = plotyy(concSmallIO,IOPS,concSmallIO,latency,’plot’); axes(haxes(1)); xlabel(‘Outstanding Small IO’) ylabel(‘IOPS’); title(‘Random I/O’,’FontSize’,14) j = 1; [m n]=size(latency) while latency(j) < 20     j = j + 1;     if n == j,         break;     end end   text(concSmallIO(j-1),IOPS(j-1),[int2str(IOPS(j-1)),’ IOPS with ‘,num2str(latency(j-1)),’ ms latency \rightarrow’ ],’HorizontalAlignment’,’right’,’FontSize’,12) axes(haxes(2)); ylabel(‘Latency(ms)’);

sequentialIOChart.m

 function sequentialIOchart(largeIO,mbps) plot(largeIO,mbps,’-rs’,’LineWidth’,2,… ‘MarkerEdgeColor’,’k’,… ‘MarkerFaceColor’,’g’,… ‘MarkerSize’,10) xlabel(‘Outstanding Large IO’) ylabel(‘MB/s’) title(‘Sequential I/O’,’FontSize’,14)

orion_chart_basic.m

 %Use this file for ORION test output executed with -run basic option %When the directory selection pop appears, pick the folder that contains %ORION run outputs. orion_directory = uigetdir();  latency_file = dir(fullfile(orion_directory,’*_lat.csv’)); iops_file = dir([orion_directory,’\*_iops.csv’]); mbps_file = dir([orion_directory,’\*_mbps.csv’]);  latency = csvread(fullfile(orion_directory,latency_file.name), 1, 1) latency = latency(1,:) small = csvread(fullfile(orion_directory,latency_file.name), 0, 1); small = small(1,:)   iops = csvread(fullfile(orion_directory,iops_file.name), 1, 1); iops = iops(1,:);   subplot(2,1,1); randomIOchart(small, iops, latency)   mbps = csvread(fullfile(orion_directory,mbps_file.name), 1, 1); large = csvread(fullfile(orion_directory,mbps_file.name), 1, 0); large = large(:,1); subplot(2,1,2); sequentialIOchart(large,mbps);

orion_chart_detailed.m

 %Use this file for ORION test output executed with -run detailed option %When the directory selection pop appears, pick the folder that contains %ORION run outputs. orion_directory = uigetdir();  latency_file = dir(fullfile(orion_directory,’*_lat.csv’)); iops_file = dir([orion_directory,’\*_iops.csv’]); mbps_file = dir([orion_directory,’\*_mbps.csv’]);  latency = csvread(fullfile(orion_directory,latency_file.name), 1, 1); small = csvread(fullfile(orion_directory,latency_file.name), 0, 1); small = small(1,:); large = csvread(fullfile(orion_directory,latency_file.name), 1, 0); large = large(:,1);   subplot(2,2,1);surf(small,large,latency);xlabel(‘Outstanding Small IO’);ylabel(‘Outstanding Large IO’);zlabel(‘Latency (ms)’);   iops = csvread(fullfile(orion_directory,iops_file.name), 1, 1); subplot(2,2,2);surf(small,large,iops);xlabel(‘Outstanding Small IO’);ylabel(‘Outstanding Large IO’);zlabel(‘IOPS’);   mbps = csvread(fullfile(orion_directory,mbps_file.name), 1, 1) small = csvread(fullfile(orion_directory,mbps_file.name), 0, 1); small = small(1,:) large = csvread(fullfile(orion_directory,mbps_file.name), 1, 0); large = large(:,1) subplot(2,2,[3 4]);surf(small,large,mbps);xlabel(‘Outstanding Small IO’);ylabel(‘Outstanding Large IO’);zlabel(‘MB/s’);

Welcome to Microsoft Office 2007

Since my new company Krom Technology Services A.Ş. is a Microsoft partner, I had the chance to experience new Office Enterprise 2007. The only thing can be said about new Office is that Microsoft will never bankrupt unless it gives up Office.

New Office not only brings new applications like Microsoft Groove 2007 (http://office.microsoft.com/assistance/asstvid.aspx?assetid=XT100627131033&vwidth=700&vheight=530&type=flash&CTT=11&Origin=HA101672641033 ), but also Microsoft completely changes how the Office looks. It has to be related with new Vista environment most probably. Old menu bar (File, View, etc.) has been removed and new marshmallow style user menus are presented instead. You can find whatever you search immediately. Moreover you can customize them as you like.

One thing that really beats me is new equation editor Microsoft Office 2007 provides. You don’t need to use Equation Editor 3.0 explicitly any more. You just press Alt + = while you are writing your text and equation editor is opened for you and you embed whatever equation you like into your work.

Other good news is for Adobe Acrobat Professional users. You don’t need Adobe Professional any more in order to convert Office documents into Portable Document Files (pdf). That’s because Office 2007 brings built-in support for pdf and XML Paper Specification (XPS).

Another good thing is Microsoft really improves it crash recovery structure for Office applications. You remember nasty crashes while you are writing you projects. Right? New Office 2007 has a more strong resistance against document corruption.

Besides virtual changes file extensions are also changed in Office 2007. Good old Word documents have docx extensions, Excel files have xlsx extensions. As you may understand all old file extensions are padded with an x letter. Older file Office files can easily be opened but in order to use new features like this new equation editor you need to convert you files to this new format.

One good new is for bloggers, I have prepared this blog post in Office 2007 Word and just click on some buttons to publish it. There exist build-in supports for various blog accounts like wordpress, BlogSpot, etc.

Something came weird to me is that new Office 2007 is not any slower than Office 2003 J

Those are my a few experiences on Microsoft Office 2007 (especially Word, Excel). I have written my senior project report using Microsoft Office 2007 (I can hear the ones saying “This was a risk”J) and convert it to pdf and I will be submitting it tomorrow. It was a fast and good job with the help of Microsoft Office 2007.

You know me. I’ve never been a Microsoft guy, except that I like working with Office. And this new Office is really good.

My master has just moved…

My Yoda has just moved to WordPress in this week. The only thing left to me is to perform the ritual. Following his foot steps…