How to install Vagrant with Oracle® VM VirtualBox and libvirt support on Antergos Linux

Installation

I recently needed to install Vagrant with support for Oracle® VM VirtualBox and libvirt for my dixie project at GitHub. Some of the virtualbox dependencies was found in this excellent post over at forum.antergos.com. Then I wrapped it into a script, and the here is the part of the script that installs and configures it:

if [ $(uname -r | grep ARCH | wc -l) -gt 0 ]; then

  sudo pacman -S --needed --noconfirm vagrant

  sudo pacman -S --needed --noconfirm libvirt

  sudo pacman -S --needed --noconfirm linux-headers

  sudo pacman -S --needed --noconfirm virtualbox virtualbox-guest-iso

  sudo pacman -S --needed --noconfirm vde2 net-tools virtualbox-ext-vnc virtualbox-host-modules-arch

  sudo pacman -S --needed --noconfirm ansible rsync

  sudo su -c "modprobe vboxdrv" || echo "Reboot your computer and try again"

  if [ ! -f "/etc/modules-load.d/virtualbox.conf" ]; then

    sudo touch /etc/modules-load.d/virtualbox.conf

  fi

  if [ $(grep vboxdrv /etc/modules-load.d/virtualbox.conf | wc -l) -eq 0 ]; then

    sudo su -c 'echo "vboxdrv" >> /etc/modules-load.d/virtualbox.conf'

  fi

  if [ $(grep vboxnetadp /etc/modules-load.d/virtualbox.conf | wc -l) -eq 0 ]; then

    sudo su -c 'echo "vboxnetadp" >> /etc/modules-load.d/virtualbox.conf'

  fi

  if [ $(grep vboxnetflt /etc/modules-load.d/virtualbox.conf | wc -l) -eq 0 ]; then

    sudo su -c 'echo "vboxnetflt" >> /etc/modules-load.d/virtualbox.conf'

  fi

  if [ $(grep vboxpci /etc/modules-load.d/virtualbox.conf | wc -l) -eq 0 ]; then

    sudo su -c 'echo "vboxpci" >> /etc/modules-load.d/virtualbox.conf'

  fi

fi

weblogic.store.PersistentStoreFatalException: [Store:280019]There was an error while writing to a storage

The problem

We discovered that an Oracle® WebLogic managed server was in status FAILED on a development environment. The apparent reason for this was the following error:

weblogic.store.PersistentStoreFatalException: [Store:280019]There was an error while writing to a storage

The cause

The can be multiple reasons or the above error, but the logical guess was that our /data mount point had run out of disk space, because that mount point was where our domain configuration (and persistent stores) were saved. That theory turned out to be wrong. The /data mount point was only 12% utilized on a 100GB partition, so there was plenty of available disk.

Then I tried to create an empty file on the mount point:

# touch /data/dummy

# touch: cannot touch `/data/dummy': Read-only file system

When checking the /var/log/messages file, I found the reason for that:

May 11 07:37:45 <hostname> kernel: JBD2: I/O error detected when updating journal superblock for dm-25-8.

May 11 07:37:45 <hostname> kernel: EXT4-fs error (device dm-25): ext4_journal_start_sb: Detected aborted journal

May 11 07:37:45 <hostname> kernel: EXT4-fs (dm-25): Remounting filesystem read-only

Comparing the commands cat /etc/fstab and ls -l /dev/mapper/ | grep dm-25 also confirmed that device dm-25 was indeed the source for the /data mount point.

The solution

After shutting down everything as cleanly as I could, I checked if there were still processes trying to use the /data mount point:

lsof | grep "/data"

It turned out that there was a few processes I was unable to shutdown cleanly, so I had to kill those. 

Then I unmounted the /data mount point:

umount /data

Finally I fixed the issue by running fsck and mounting the partition again. 

Caution: Running fsck on a partition can in worst case scenario be destructive, so make sure you have a valid backup!

fsck /data

mount /data

Now I could start up Oracle® WebLogic Server without any issues.

RDA-10913: Unknown profile "FM12c_SoaMin" when running Remote Diagnostic Agent on Oracle® SOA Suite 12.1.3

The problem

You are trying to collect information about the Oracle® SOA Suite 12.1.3 using the Remote Diagnostic Agent (RDA), but it fails with a RDA-10913: Unknown profile "FM12c_SoaMin" error, like shown below:

source $DOMAIN_HOME/bin/setDomainEnv.sh

cd $SOA_ORACLE_HOME

cd ../oracle_common/rda

./rda.sh -s soa_issue -p FM12c_SoaMin

RDA-00013: Error reported by the command "setup":

 RDA-00014: Request error "Setup":

  RDA-10913: Unknown profile "FM12c_SoaMin"

The cause

There is something wrong with the RDA supplied with  Oracle® SOA Suite 12.1.3, and this is how I discovered that:

The command ./rda.sh -V told me I was running RDA 8.02, which looked fine.

By reading the full output from ./rda.sh -V I could also see that the required modules where installed, according to Oracle Support Doc ID 391983.1.

Using ./rda.sh -h told me which options RDA is accepting, And in particular I noted the -L flag:

-L     List the available modules, profiles, and conversion groups

./rda.sh -L | grep Soa

  FM11g_Soa                        Oracle SOA Suite 11g problems

  FM11g_SoaMin                     Oracle SOA Suite 11g problems (minimum)

As you can see, the RDA shipped with Oracle® SOA Suite 12.1.3 was only supporting Oracle® SOA Suite 11g diagnostics.

The solution

1. Downloaded the latest version of the RDA tool for Oracle® Fusion Middleware. At the time of writing this, the patch number was 22783073.
2. Used opatch apply to install the patch I just downloaded.
3. Verified which SOA profiles were now available by running:
   ./rda.sh -L | grep Soa
   OFM_SoaMax                       OFM SOA Suite problems (11g/12c+) : full
   OFM_SoaMin                       OFM SOA Suite problems (11g/12c+) : minimum
    
4. Ran the RDA with the new module for Oracle® SOA Suite 11g and 12c:
   ./rda.sh -s soa_issue -p OFM_SoaMin

Error "ORA-01882 timezone region not found" when creating new domain in Oracle® WebLogic

The problem

During domain configuration I received an error saying ORA-01882 timezone region not found when I clicked on the Get RCU Configuration button.

The cause

This is because your operating system is either using a invalid timezone value, or using the Etc/UTC or Universal timezone. The two latter examples are due to a bug when connecting to the database. 

The solution

1. Exit the domain configuration assistant
2. Set a timezone value accepted by the database: TZ=UTC; export TZ
3. Start the domain configuration assistant again by running config.sh
4. To prevent the same issue from happening again (assuming you hit the bug), you should add the corrected TZ to the software owner's profile, e.g. in $HOME/.bash_profile if you are using the Bourne Again Shell. However, if your Linux server clock is configured with an invalid timezone, you should fix it at Linux level by using a IANA compliant timezone.

OPatch failed with error code 73

The problem

I came across a sandbox server which had an interesting problem with oraInventory. When running opatch I received the following error:

Inventory load failed... OPatch cannot load inventory for the given Oracle Home.

Possible causes are:

   Oracle Home dir. path does not exist in Central Inventory

   Oracle Home is a symbolic link

   Oracle Home inventory is corrupted

LsInventorySession failed: Oracle Home inventory cannot be loaded.

OPatch failed with error code 73

The cause

The immediate cause was that /u01/app/oraInventory was empty besides the "locks" directory that got created when I ran $ORACLE_HOME/OPatch/opatch lsinventory.

/etc/oraInst.loc pointed to the /u01/app/oraInventory directroy, and so did $ORACLE_HOME/oraInst.loc.

I looked for other copies of oraInventory on the server, but there were none.

I can only speculate on why /u01/app/oraInventory was empty. Perhaps someone deleted the contents by mistake, or by some misguided effort to free up disk space.

The solution

Had this been a production system it would have made sense to check the file system backup, and do a restore of /u01/app/oraInventory. But his was just a sandbox, and no file system backup existed. 

The solution was to run $ORACLE_HOME/oui/bin/attachHome.sh to populate the oraInventory again. 

Now the command $ORACLE_HOME/OPatch/opatch lsinventory worked fine.

How to find which Java thread on WebLogic that causes high CPU utilization on Linux

Scope

Sometimes your Linux server is having close to 100% CPU utilization. It is easy and quick to find which process is spending more CPU by running the top command. It takes a few more steps to establish which Java(TM) thread is using CPU, and what that thread is doing. This blog will explain the procedure for gather evidence to your developers that their code is too CPU intensive.

Scenarios

I have tried this procedure using the following versions of Oracle® WebLogic and Java(TM):

• Oracle® WebLogic 12.1.3 with Oracle® Java(TM) SE 7 - jdk1.7.0_79
• Oracle® WebLogic 10.3.6 with Oracle®Java(TM) SE 7 - jdk1.7.0_79
• Oracle® WebLogic 10.3.6 with Oracle® Java(TM) SE 6 - jdk1.6.0_45
• Oracle® WebLogic 10.3.6 with Oracle® JRockit(R) 6 - R28.2.7

These guidelines would also work on other servers using Oracle® Java(TM) SE.

Overview

The procedure for finding the most CPU intensive thread is basically the same for each combination of Oracle® WebLogic and Java(TM):

• Find the ID of the process that has highest CPU utilization
• Find the ID of the thread that has highest CPU utilization based on the process ID found above.
• Take a thread dump of the process based on the process ID found above.
• Search for the thread ID in the thread dump
• Identify the code that causes issues

Find the ID of the process that has highest CPU utilization

Use the top command and look at the CPU% column to identify which PID has highest CPU utilization. In the case below, the PID with 7253 uses most CPU%.

Find the ID of the thread that has highest CPU utilization based on the process ID found above

Again we can use the top command to find out which threads are spending most CPU% with the PID as input. The command will be:

top -H -p <PID>

Take a thread dump of the process based on the process ID found above

Use the jstack <pid> command with Java HotSpot(TM) or the jrcmd <pid> print_threads command with Oracle® JRockit(R) to get a thread dump. For instance:

$JAVA_HOME/bin/jstack 7253 > /tmp/stack.txt

Search for the thread ID in the thread dump file

Some tools will write the thread IDs as decimal numbers and others will write it as hexadecimal numbers. In this blog post I used Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode) which wrote the thread IDs as hexadecimal numbers. When testing Oracle® JRockit(R) however, my thread IDs were written with decimal numbers. 

From the list of thread IDs above, I converted the three thread IDs with highest CPU% to hex:

[oracle@localhost dom_12c_hotspot7]$ printf "%x\n" 7255
1c57
[oracle@localhost dom_12c_hotspot7]$ printf "%x\n" 7286
1c76
[oracle@localhost dom_12c_hotspot7]$ printf "%x\n" 7291
1c7b

I can now search for 1c57, 1c76 and 1c7b in the /tmp/stack.txt file I created.

Nothing exciting about 1c57 as it was only the "VM Thread":

The other thread ID 1c76 is coming from my custom code:

The third thread ID 1c7b is also coming from my custom code, and that thread is even STUCK.:

This was just an example on how to find threads with high CPU utilization. The code deployed was written poorly on purpose to cater for this article. Finding the performance issue on a real life application will probably prove to be more complex. In case there are multiple threads that fights over the CPU, then it is probably better to do live analyses using a tool like JVisualVM.

The story of my journey to Linux

The journey

While I have been working on both Windows, UNIX and Linux since 1997, it was only a few years ago that I started using Linux as my primary operating system at home.

During my professional career I have worked on both HP Compaq Tru64, IBM AIX, HP/UX, Sun Solaris (both Sparc and Intel), SuSE Linux, Red Hat Enterprise Linux, CentOS and Oracle Linux.

My first encounter with computers was when a few of my friends got an Amiga 500 in the late 80'es. About the same time I got a Commodore 64, and while these machines were mainly used for gaming, it was also my first encounter with programming. In BASIC.

Around 1990 my father purchased an Atari 1040ST, which is used to create music. He also owned a PC AT where I learned word processing. 

In 1993 I took a few classes at an engineering college, where one of the perks was your own personal shell account. I do not know which version of UNIX it was, and I mainly used it for email and accessing Bulletin Board Systems. Either the same year, or the year after, I bought my first PC. It was an Olivetti with Windows 3.1. I do not recall what happened to it, but I did not have it for a long time. 

In 1996 I got a shell account at the university, and as far as I can remember they were running Sun Solaris. The same year I bought my second PC, which was assembled by a PC builder hobbyist. It came with a boot virus, so I got a crash course in the format and fdisk commands before I finally got the problem solved and Windows 95 reinstalled. This ordeal made me look for other options, and during the next few weeks I obtained copies of both OS/2 and Red Hat Linux. The former was never able to run stable on my machine, but the latter had potential as I was already used to working in the shell. In the end though, I installed Windows 95 yet again.

Since the late 90'es I have been using the Internet regularly. At first it was through a modem 28.8 kbps and then through ISDN 64 kbps or bundled up to 128 kbps. Finally around 2002-2003 I got a DSL connection, and then it occurred to me that I could host a server at home. At that time I always had a few spare computers, so I bought a bundle with SuSE Linux and written documentation. While SuSE worked fine, I was more comfortable with Red Hat, because most my professional encounters with Linux was with Red Hat Linux. In 2003, Red Hat had discontinued their free Red Hat Linux distribution, and only offered a paid option. Instead they started the Fedora Core project based on the source code from Red Hat Linux 9.

For the next 5 years I was running Fedora Linux on my home server(s), while keeping Windows on the desktop. The only reason I kept Windows was because I enjoyed playing a few computer games that only would run on Windows. 

In 2008 I did not longer have time to manage home servers, and I had also begun taking an interest in OS X. First I bought a cheap Mac Mini to check if OS X was anything for me. I really liked it, so I bought a iMac to replace my aging desktop PC. Later the same year I also bought a Macbook Pro. 

Between 2009 and 2011 I was using Windows on my gaming computer and OS X for surfing, programming and word processing. 

By 2012 I was no longer playing computer games, and I had given away my gaming computer to a friend of the family. The aging Macbook Pro was still catering for my surfing and word processing needs, and since I was quite satisfied with OS X, I saw no reason so use Linux at home during these years.

In 2014 my Macbook Pro was now 6 years old, and beginning to feel slow. The question was if I should buy a new Macbook Pro, or get a cheaper PC and install Fedora Linux on it. Having a PC with Windows 8 was no option for me. I do have a few gadgets, mainly HR watches, which only have drivers and software for Windows and OS X. In the end I decided to buy a Lenovo Thinkpad E540. I installed Windows XP in VirtualBox to cater for my gadgets.

I have always been a huge fan of the Red Hat family of Linux distributions. For many years I have enjoyed using Fedora because it is a leading edge, while still managing to be quite stable. As I have grown older and have gotten less time to spend in front of the computer, I wished I did not have to upgrade my Fedora installation twice a year, or do a reinstall. So I started looking for options. I know quite a few people that uses Linux Mint, and they are very happy with it, since both Cinnamon, XFCE and KDE uses "classic" menus similar to the Start button menu in Windows. As Gnome3 user I found Ubuntu 14.04 LTS with Unity more welcoming. I also ran into several technical issues with Linux Mint which I never had with neither Fedora or Ubuntu. Wifi instability being the major issue. 

I have been very satisfied with Ubuntu 14.04 LTS. It has been very stable without any technical issues at all. Pretty much all  the software I have needed have been available in the official repositories, including audio and video playback support. I can recommend Ubuntu to others. It just works. At least on my hardware :)

It was more a philosophical issue that I kept looking for an alternative. I wanted to use a distro that "just works", was community driven, active development, was leading edge but without a need to reinstall, and that had Gnome3 as an official desktop environment. Manjaro seemed like an obvious choice. While Gnome3 was not an official DE, it could either be installed as a community edition, through the repositories in the offical editions. Manjoro is a really beautiful and polished distro, and I would have chosen it, if it hadn't been that I discovered Antergos.

Antergos Linux is a rolling release distro based on Arch. It is leading edge, and supports several of the leading desktop environments including Gnome3. It also looks great as it is using Numix icons. I installed it, and instantly fell in love. I had several technical issues with the LightDM display manager, in particular when awakening from hibernate. I believe this issue was caused by my hardware, so after failing to get LightDM to work properly, I simply disabled and uninstalled LightDM and installed and enabled GDM instead.

I am now caught between Antergos Linux and Fedora. Antergos Linux has it's lightweight and rolling appeal, but Fedora is part of the Red Hat family that I love so much. Linux offers great diversity, and I embrase it.