PyCharm 2019.2 IDE was not starting on MacOSX: missing jdk

PyCharm 2019.2  is looking for java8 and does not come with a jdk folder. So if you have java11 it will not even start. I have had this problem with the new PyCharm installed from the ToolBox, but also with the new manually installed

Solution:
Either install java8 or if you have a working app from JetBrains like a previous manual installation you can copy the jdk from there and paste itin your new copy.

The ToolBox apps are installed in

~/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-0/192.6262.63/PyCharm.app

In order to copy the previous jdk from a working pycharm (or other JetBrains app) you can navigate to a manually installed old app in "/Applications" and right click in the app and select "Show Package Content". In the "Contents" folder there would be a "jdk" folder. Copy it to your not working JetBrains app.


REFS:
https://intellij-support.jetbrains.com/hc/en-us/community/posts/360003340039-PyCharm-2019-1-won-t-open-through-Toolbox-App

Still a problem:

PyCharm2019.2 does not open the terminal (OSX 10.10.5)

Cannot open Local Terminal

Failed to start [/bin/bash, --rcfile, /Applications/PyCharm.app/Contents/plugins/terminal/jediterm-bash.in, -i]

Change Docker default software dir

Change Docker default software dir

For Docekr 17.xx in Ubuntu 16.04 don't follow the advice of updating the /etc/default/docker and change -g option in DOCKER_OPTS. This advice is for OLD debian using upstar system. If your linux uses Systemd do the following: use the   /etc/docker/daemon.json and add a graph directive

{
    "graph": "/mnt_docker/docker_files"
}

XRDP in centos7: installation, enabling it and setting port for reconnecting a previous session

RDP is more efficient and has better performing than VNC and probably is more secure.

Installation

The adventage of x11vnc is that you have the same desktop that is open in your workstation and two persons can share the screen. The cons is that you need to have a logged user in the Xs before using it. If you restart the machine then you need to do some tricks to us x11vnc as root to be able to go to the splash window.

XRDP is better but by default you open different sessions each time and is not trivial how to get the current session active in your workstation.

Anyway, if you use Centos7 you will have probles usig XRDP because there are some issues with selenux (the security system for applications in redhat)

XRDP can be installed from EPEL repo (the essential extra packages).

# install EPEL repo
$ sudo yum install epel-release
$ sudo yum update

# install XRDP
$ sudo yum install xrdp tigervnc-server

$ sudo service xrdp start 

$ sudo service xrdp-sesman start

After installing, activate the service

 $ sudo systemctl enable xrdp.service

Confirm that XRDP is runing

# netstat -antup | grep xrdp
tcp        0      0 0.0.0.0:3389            0.0.0.0:*               LISTEN      1508/xrdp
tcp        0      0 127.0.0.1:3350          0.0.0.0:*               LISTEN      1507/xrdp-sesman

But the job is not finished. By default XRDP listen to 3389 but this service is not allowed to listen to this port by default. You need to fix security and firewall to allow XRDP listen to 3389 (selenux) and allow external connections to 2289 (firewall)

# allow selemux (use one of the two options. Option 2 is prefered) 
## option 1
$ sudo chcon --type=bin_t /usr/sbin/xrdp
$ sudo chcon --type=bin_t /usr/sbin/xrdp-sesman

## option 2
$ sudo semanage fcontext -a -t bin_t /usr/sbin/xrdp
$ sudo semanage fcontext -a -t bin_t /usr/sbin/xrdp-sesman
$ sudo restorecon -v /usr/sbin/xrdp*

# using chcon updates the SELinux context temporarily until the next relabelling.
# A permanent method is to use semanage and restorecon


# allow firewall for external connections (just in case is not open by default)
$ sudo firewall-cmd --permanent --zone=public --add-port=3389/tcp
$ sudo firewall-cmd --reload 

Configuration

The config file is in /etc/xrdp/xrdp.ini
The port by default is 3389 and you can change here:

[globals]
# xrdp.ini file version number
ini_version=1

bitmap_cache=yes
bitmap_compression=yes
port=3389

And the setting for reconnecting to the previous session is in section "[XRDP1]" for port attribute. Set to ask-1

[xrdp1]
name=sesman-Xvnc
lib=libvnc.so
username=ask
password=ask
ip=127.0.0.1
port=ask-1
delay_ms=2000

ask would ask you for a port in the log in. If no port given, with the -1 a new one is assigned

If you can remember the current port for your session see next section

How to find the current XRDP session port

You could login a ssh session and find out the number by

netstat -tulpn | grep vnc

and you will get something like the following

tcp        0      0 127.0.0.1:5910          0.0.0.0:*               LISTEN      5365/Xvnc

and then you know 5910 was the port you connected to.



More infor about XRDP and X11vnc and vnc links can be found at http://c-nergy.be/blog/?p=6063

Choosing the right dark theme in R studio

Rstudio Dark Themes

Seems that Iddle fingers, Mervibore soft,Twilight and Tomorrow Night have the softest color schema with good contrast for comments and not to bright colours for the long coding nights ahead. I like the comments in italic and with not a prominent color (I want to keep my attention at the code) so Twilight is my current chose.





All dark themes in R-studio


 
 
 
 
 
 
 
 
 

Installing mysqlworkbench and mysql utils in MacOSX Yosemite

Installing MySQL, mysqlworkbench and mysql-utils in Yosemite

For installing things like *nix in MacOSX I use homebrew.

Homebrew is fantastic for installing tar.gz files in textual way, but there are some packages that are only in .dmg format and you need to use a package manager that knows how to install this GUI based installers. This package manager is homebrew cask (caskroom project)

First you need to install mysql

$ brew install mysql
$ mysql.server start 
# and optionaly make it more secure
$ mysql_secure_installation

Then you need to 'tap' the cask project and then get your 'cask' before 'pouring' it.

$ brew tap caskroom/cask
# optional
$ brew install homebrew/completions/brew-cask-completion

## install workbench and utils

$ brew cask install mysqlworkbench
# mysqlworkbench staged at '/opt/homebrew-cask/Caskroom/mysqlworkbench/6.3.6'
# Symlinking App 'MySQLWorkbench.app' to '/Users/pmg/Applications/MySQLWorkbench.app'

$ brew cask install mysql-utilities
# mysql-utilities staged at '/opt/homebrew-cask/Caskroom/mysql-utilities/1.5.6'

Note that workbench is symlinked to your home apps not the main Application folder

~/Applications/MySQLWorkbench.app

Launch it and you will get this window with the MySQL Utils icon at the top right

When you click in the MySQL Utilities.... OPS!! it does not work!

MySQL Utils launch a console and the error message is

$ mysqluc -e "help utilities"
ERROR: The MySQL Connector/Python module was not found. MySQL Utilities requires the connector to be installed. Please check your paths or download and install the Connector/Python from http://dev.mysql.com.

OK, let's find the connector googleing it. And find it

[EDIT] I have just missed that the mysql-connector-python is also in cask and you can install with `brew cask install mysql-connector-python` BUT IT DID NOT WORK FOR ME

$ brew search mysql

automysqlbackup                    mysql ✔                            mysql++                            mysql-cluster                      mysql-connector-c                  mysql-connector-c++                mysql-sandbox                      mysql-search-replace               mysqltuner                       

homebrew/php/php53-mysqlnd_ms                       homebrew/php/php55-mysqlnd_ms                       homebrew/versions/mysql51                           homebrew/versions/mysql56                           Caskroom/cask/mysql-utilities                       Caskroom/cask/navicat-for-mysql                   

homebrew/php/php54-mysqlnd_ms                       homebrew/php/php56-mysqlnd_ms                       homebrew/versions/mysql55                           Caskroom/cask/mysql-connector-python                Caskroom/cask/mysqlworkbench                      

$ brew cask install mysql-connector-python

Error: No available Cask for mysql-connector-python

Error: nothing to install

at http://dev.mysql.com/downloads/connector/python/ but instead of selecting Mac choose "platform independent"

choose download and and then copy the link from "no thanks, just start download"

Go back to your shell and download it (you can do it in the mysql-utils dir just to have all together)

$ cd /opt/homebrew-cask/Caskroom/mysql-utilities/
$ wget http://dev.mysql.com/get/Downloads/Connector-Python/mysql-connector-python-2.1.3.tar.gz
$ tar -zxvf mysql-connector-python-2.1.3.tar.gz
$ cd mysql-connector-python-2.1.3/

# install for both python
$ sudo python ./setup.py install
$ sudo python3 ./setup.py install
</ pre>

Now when you click in the MySQL-Utils you have a working connection and you see this in the launched terminal

P.S.

I have tried to install the connector from python using pip but it did not work due a problem with ConfigParser. I did not investigate it further because I have already a working solution.

        MacBook-Pro:[~]

        $ pip3 install MySQL-python

        Collecting MySQL-python

          Downloading MySQL-python-1.2.5.zip (108kB)

            Complete output from command python setup.py egg_info:

            Traceback (most recent call last):

              File "<string>", line 20, in <module>

              File "/private/var/folders/m0/spbgfzvd17jdfqgx90w1_hx40000gn/T/pip-build-0nif_qmi/MySQL-python/setup.py", line 13, in <module>

                from setup_posix import get_config

              File "/private/var/folders/m0/spbgfzvd17jdfqgx90w1_hx40000gn/T/pip-build-0nif_qmi/MySQL-python/setup_posix.py", line 2, in <module>

                from ConfigParser import SafeConfigParser

            ImportError: No module named 'ConfigParser'

            ----------------------------------------

        Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/m0/spbgfzvd17jdfqgx90w1_hx40000gn/T/pip-build-0nif_qmi/MySQL-python

        You are using pip version 7.1.2, however version 8.0.2 is available.

        You should consider upgrading via the 'pip install --upgrade pip' command.

       

        MacBook-Pro:[~]

        $ pip3 install ConfigParser

        Collecting ConfigParser

          Downloading configparser-3.3.0r2.tar.gz

            Complete output from command python setup.py egg_info:

            Traceback (most recent call last):

              File "<string>", line 3, in <module>

              File "/usr/local/lib/python3.5/site-packages/setuptools/__init__.py", line 5, in <module>

                import distutils.core

              File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/core.py", line 18, in <module>

                from distutils.config import PyPIRCCommand

              File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/config.py", line 7, in <module>

                from configparser import ConfigParser

              File "/private/var/folders/m0/spbgfzvd17jdfqgx90w1_hx40000gn/T/pip-build-0ouo8j8z/ConfigParser/configparser.py", line 397

                _KEYCRE = re.compile(ur"%\(([^)]+)\)s")

                                                     ^

            SyntaxError: invalid syntax

            ----------------------------------------

new security vulnerability called POODLE

There Is a New Security Vulnerability Named POODLE, and It Is Not Cute: http://www.wired.com/2014/10/poodle-explained/

https://supportforums.cisco.com/discussion/12326341/sslv3-poodle-vulnerability

http://www.zdnet.com/google-reveals-major-flaw-in-outdated-but-widely-used-ssl-protocol-7000034677/

POODLE affects SSLv3 or version 3 of the Secure Sockets Layer protocol, which is used to encrypt traffic between a browser and a web site or between a user’s email client and mail server. It’s not as serious as the recent Heartbleed and Shellshock vulnerabilities, but POODLE could allow an attacker to hijack and decrypt the session cookie that identifies you to a service like Twitter or Google, and then take over your accounts without needing your password.

According to the team's Bodo Möller: "This vulnerability allows the plaintext of secure connections to be calculated by a network attacker."

This is a client security hole. This is a vulnerability in the old SSLv3 (relaying in 80's code), but even if you have all set for using TLS, in many applications the server can trick you to downgrade to SSL and then it can stole you some security cookies.

This vulnerability is a risk in public wifi but not at home or work.

------------------

Mozilla says that it is making Firefox 34 safe from POODLE by disabling SSL 3.0 by default. The code which does this is already baked into the Nightly channel, and will make its way to the Aurora and Beta channels also "in the next few weeks".

----------
MicroSoft Advisory and Workarounds:

https://technet.microsoft.com/en-us/library/security/3009008.aspx
This workaround involves Group Policy Editor. For those using versions of Windows that don't include GPE simply go to Control Panel/Internet Options/Advanced/Security and uncheck the box for SSL 3.0 (SSL 2.0 should already be unchecked).

Ubuntu 14.04 new Ibus is interfering with default emacs keybinding control-space

In emacs you select text marking a starting selection point with control-space. This is a fundamental key binding in emacs. Now in the new version of Ubuntu (14.04) the default service for controlling multilingual input in linux has been changed to Ibus. The issue is that Ibus default key binding to control-space.

If you want to use emacs you probably should change this to something else. Go to a terminal and write 'ibus-setup' and at the right of the input method line you will find three dots '...' click there and change your key binding to Ibus pressing in the three dots in the key code line and then clik in disable. The text will change to 'new accelerator', press your combination of keys and you are done. I have changed mine to super-alt-space (super is the windows key).

perl gzip libraries (probably zlib issue) does not play well with bgzip files.

In bioinformatcis, bgzip files are important for random access to big files . Bgzip is a program modified from gzip program that uses block compression and is fully backwards compatible with gzip. But I have issues when using bgzip compressed vcf files with Perl scripts that uses IO::Uncompress::Gunzip (that I believe it uses zlib under the hood). A similar problem happen to my recently with snpeff program (Java). In both cases the data is decompressed but truncated after a few hundred lines aprox. I could be totally wrong but I was wondering if zlib (or whatever gzip compatible library they are using) is getting confused with the bgzip bloks and only processing one or a few of them leaving the output incomplete. perl code that does not work:

#!/usr/bin/env perl
use strict;
use IO::Uncompress::Gunzip qw(gunzip $GunzipError) ;

my $infile = shift;
my $infh = IO::Uncompress::Gunzip->new( $infile ) 
         or die "IO::Uncompress::Gunzip failed: $GunzipError\n";
my $line_count = 0;
while (my $line=<$infh>){

    $line_count++
}
print "total lines read = $line_count\n";

This gives 419 lines

    $ perl /home/pmg/tmp/test_zlib-bgzip.pl varsit.vcf.gz
    total lines read = 419

but using open with gzip pipe works:

    #!/usr/bin/env perl
    use strict;
    # I can use bgzip intead gzip
    my $infile = shift;

    open(my $infh , 'gzip -dc '.$infile.' |'); 
    my $line_count = 0;
    while (my $line=<$infh>){

        $line_count++
    }
    print "total lines read = $line_count\n"; 

Gives the expected number of lines

    $ perl /home/pmg/tmp/test_gzip-bgzip.pl varsit.vcf.gz
    total lines read = 652829

I googled about and I was unable to find quickly any relevant entry, but this is something that I am sure other people would have already faced. Do someone have a clue about why is this happening? I am using ubuntu 12.04.4 with perl 5.16

[UPDATE 2014-02-28]: finally a clue come from biostars where Heng Li remind me a footnote in the SAM specs about a java library for gzip that only sees first block of bgzip when decompressing. Seems that Perl gzip implementations had the same problem.

automatic encrypting decrypting with emacs

I have been for long time wondering if there was a way to automatically encrypt and decrypt files when accessing them, not leaving an unencrypted file around.

Today I have an epiphany: "let's see if emacs can do that". As an emacs power user I felt as an idiot. Emacs has in the core this ability since emacs 23!! and I never realized about it :-(.

Emacs has incorporated the EasyPG code into its core.
http://www.emacswiki.org/emacs/EasyPG

 Things can not be more easy. Put in your .emacs

(require 'epa-file)
(epa-file-enable)

;; as it is annoying to be asked if I want passphrase or publick/private key
;; I set passphrase as default
(setq epa-file-select-keys nil)

I also would like to have a gpg agent cache. In ubuntu 12.04 the package is called gnupg-agent

sudo apt-get install gnupg-agent
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  libassuan0 pinentry-gtk2
Suggested packages:
  pinentry-doc
The following NEW packages will be installed:
  gnupg-agent libassuan0 pinentry-gtk2
0 upgraded, 3 newly installed, 0 to remove and 1 not upgraded.  

Now you only need to open a file ended with .gpg and automatically emacs would ask you for the pasword for decryption and again for saving. No unencrypted temporary files are stored, or at least I am not aware of them.

Reducing batch effects in methylation analysis

In our sequencing facility we have processed 4 illumina 450K methylation chips for the same study at 2 diferent times. We took care of spreading case and controls among all the chips. We can see a clear time-of-processing batch effect:

The chips were run in two different days one month apart. first 24 samples and then the other 24 samples.



The negative controls of the second batch  have a wider range.

And wen looking at the MDS and PCA we can see that the main parameter for grouping samples is the run day (the two chips starting with 835... where done the same day and the other two another day)

I have used the nice rnbeads R package for reporting the batch effects.

Now I am exploring two methods of correcting the batch effect:

1.   limma using the batch date as block effect
• design <- model.matrix(~Block+Treatment)
  
2. ComBat in the R SVA package for correcting this

I will post the results soon. Meanwhile any comment for dealing with batch effects in 450K chips is more than wellcome.