BTRFS Snapshot Replication

This post is going to be a little bit different in terms of BTRFS and replication. The “normal” way is you use snapshots for backup purposes – or – if you use a snapshot to clone the data, you end up copying the data from the snapshot to the new location. My needs are a little bit different, so I can use the data (read-only) from the snapshot directly.

In my scenario I have a source-of-truth server with 2.3TB of data and over 800,000 files that has daily changes (additions mostly) to the data. This data change is only moderate – about 500 files/1GB per day – but the actual changes occur throughout the business day rather than at a single point of time. The original spec was to RSYNC changes to remote sites daily for the purposes of backups and distribution points, but this was quickly changed to “can we have this run every hour?” by the consumers of the data. Client systems connect to the closest server available and rsync data based on their xml payloads (not including that configuration in this post as it is out of scope).

Unfortunately RSYNC is terribly slow when copying from a single directory with many files. It gets even worse as the copy travels over SSH to geographically disconnected sites; the latency at the best site is 40ms and at the worst is over 150ms. Because this is critical data, it was deemed necessary to have MD5 checksums on each file to guarantee the distribution points are identical to that of the source of truth server. The change of data being only 500 files/1GB had little impact on the 12-20 hours it would take at our worst site.

Enter BTRFS. Yes, I know that ZFS offers this as well, but the BTRFS is slightly more native on Linux than ZFS is, and I only need the checksum (scrub) and snapshot abilities for my file systems as the data is not compressible or deduplication friendly.

Snapshot

I actually downloaded some BTRFS friendly scripts from github: https://github.com/nachoparker/btrfs-snp

chmod +x btrfs-snp
mv btrfs-snp /usr/local/sbin

btrfs-snp /mnt/btrfs/ hourly 2 3600

Sync to Remote Server

Once again, the BTRFS friendly script from github: https://github.com/nachoparker/btrfs-sync

chmod +x btrfs-sync
mv btfrs-sync /usr/local/sbin

btrfs-sync -d -v /mnt/btrfs/.snapshots/ root@10.10.3.21:/mnt/btrfs/snapshot/

My Script

In my case I needed to then utilize the data for clients to connect via RSYNC, which means the data had to be in a specific spot already advertised to those clients. Enter sym links! Here is my full script.

#ping make sure the device responds
ping -c 3 10.130.20.200

#create snapshot named 'hourly', delete any more than 2 snaps, no less than 3600 seconds old
#btrfs-snp /mnt/btrfs/ hourly 2 3600

#TESTING PURPOSES create snapshot named 'hourly', delete any more than 2 snaps, no less than 600 seconds old
btrfs-snp /mnt/btrfs/ hourly 2 600

#send the snapshot to the other system (requires authkeys ssh setup)
btrfs-sync -d -v /mnt/btrfs/.snapshots/ root@10.10.3.21:/mnt/btrfs/snapshot/

#now run the following commands on the remote system - need testing
## as this may cause rsync to fail if a player is currently loading
ssh root@10.10.3.21 'latestdir=$(ls -rt /mnt/btrfs/snapshot | tail -1) && rm /mnt/btrfs/data && ln -s /mnt/btrfs/snapshot/$latestdir/ /mnt/btrfs/data'

Other Scripts

In case they remove from github, Figured I’d put them here:

#!/bin/bash

#
# Simple script that synchronizes BTRFS snapshots locally or through SSH.
# Features compression, retention policy and automatic incremental sync
#
# Usage:
#  btrfs-sync [options] <src> [<src>...] [[user@]host:]<dir>
#
#  -k|--keep NUM     keep only last <NUM> sync'ed snapshots
#  -d|--delete       delete snapshots in <dst> that don't exist in <src>
#  -z|--xz           use xz     compression. Saves bandwidth, but uses one CPU
#  -Z|--pbzip2       use pbzip2 compression. Saves bandwidth, but uses all CPUs
#  -q|--quiet        don't display progress
#  -v|--verbose      display more information
#  -h|--help         show usage
#
# <src> can either be a single snapshot, or a folder containing snapshots
# <user> requires privileged permissions at <host> for the 'btrfs' command
#
# Cron example: daily synchronization over the internet, keep only last 50
#
# cat > /etc/cron.daily/btrfs-sync <<EOF
# #!/bin/bash
# /usr/local/sbin/btrfs-sync -q -k50 -z /home user@host:/path/to/snaps
# EOF
# chmod +x /etc/cron.daily/btrfs-sync
#
# Copyleft 2018 by Ignacio Nunez Hernanz <nacho _a_t_ ownyourbits _d_o_t_ com>
# GPL licensed (see end of file) * Use at your own risk!
#
# More at https://ownyourbits.com
#

set -e -o pipefail

# help
print_usage() {
  echo "Usage:
  $BIN [options] [[user@]host:]<src> [<src>...] [[user@]host:]<dir>

  -k|--keep NUM     keep only last <NUM> sync'ed snapshots
  -d|--delete       delete snapshots in <dst> that don't exist in <src>
  -z|--xz           use xz     compression. Saves bandwidth, but uses one CPU
  -Z|--pbzip2       use pbzip2 compression. Saves bandwidth, but uses all CPUs
  -p|--port         SSH port. Default 22
  -q|--quiet        don't display progress
  -v|--verbose      display more information
  -h|--help         show usage

<src> can either be a single snapshot, or a folder containing snapshots
<user> requires privileged permissions at <host> for the 'btrfs' command

Cron example: daily synchronization over the internet, keep only last 50

cat > /etc/cron.daily/btrfs-sync <<EOF
#!/bin/bash
/usr/local/sbin/btrfs-sync -q -k50 -z /home user@host:/path/to/snaps
EOF
chmod +x /etc/cron.daily/btrfs-sync
"
}

echov() { if [[ "$VERBOSE" == 1 ]]; then echo "$@"; fi }

#----------------------------------------------------------------------------------------------------------

# preliminary checks
BIN="${0##*/}"
[[ $# -lt 2      ]] && { print_usage                                ; exit 1; }
[[ ${EUID} -ne 0 ]] && { echo "Must be run as root. Try 'sudo $BIN'"; exit 1; }

# parse arguments
KEEP=0
PORT=22
ZIP=cat PIZ=cat
SILENT=">/dev/null"

OPTS=$( getopt -o hqzZk:p:dv -l quiet -l help -l xz -l pbzip2 -l keep: -l port: -l delete -l verbose -- "$@" 2>/dev/null )
[[ $? -ne 0 ]] && { echo "error parsing arguments"; exit 1; }
eval set -- "$OPTS"

while true; do
  case "$1" in
    -h|--help   ) print_usage; exit  0 ;;
    -q|--quiet  ) QUIET=1    ; shift 1 ;;
    -d|--delete ) DELETE=1   ; shift 1 ;;
    -k|--keep   ) KEEP=$2    ; shift 2 ;;
    -p|--port   ) PORT=$2    ; shift 2 ;;
    -z|--xz     ) ZIP=xz     PIZ=( xz     -d ); shift 1 ;;
    -Z|--pbzip2 ) ZIP=pbzip2 PIZ=( pbzip2 -d ); shift 1 ;;
    -v|--verbose) SILENT=""  VERBOSE=1        ; shift 1 ;;
    --)                shift;  break   ;;
  esac
done

SRC=( "${@:1:$#-1}" )
DST="${@: -1}"

# detect remote dst argument
[[ "$SRC" =~ : ]] && {
  NET_SRC="$( sed 's|:.*||' <<<"$SRC" )"
  SRC="$( sed 's|.*:||' <<<"$SRC" )"
  SSH_SRC=( ssh -p "$PORT" -o ServerAliveInterval=5 -o ConnectTimeout=1 -o BatchMode=yes "$NET_SRC" )
}

[[ "$SSH_SRC" != "" ]] && SRC_CMD=( ${SSH_SRC[@]} ) || SRC_CMD=( eval )
${SRC_CMD[@]} test -x "$SRC" &>/dev/null || {
  [[ "$SSH_SRC" != "" ]] && echo "SSH access error to $NET_SRC. Do you have passwordless login setup, and adequate permissions for $SRC?"
  [[ "$SSH_SRC" == "" ]] && echo "Access error. Do you have adequate permissions for $SRC?"
}

# detect remote dst argument
[[ "$DST" =~ : ]] && {
  NET="$( sed 's|:.*||' <<<"$DST" )"
  DST="$( sed 's|.*:||' <<<"$DST" )"
  SSH=( ssh -p "$PORT" -o ServerAliveInterval=5 -o ConnectTimeout=1 -o BatchMode=yes "$NET" )
}
[[ "$SSH" != "" ]] && DST_CMD=( ${SSH[@]} ) || DST_CMD=( eval )
${DST_CMD[@]} test -x "$DST" &>/dev/null || {
  [[ "$SSH" != "" ]] && echo "SSH access error to $NET. Do you have passwordless login setup, and adequate permissions for $DST?"
  [[ "$SSH" == "" ]] && echo "Access error. Do you have adequate permissions for $DST?"
  exit 1
}

#----------------------------------------------------------------------------------------------------------

# more checks

## don't overlap
pgrep -F  /run/btrfs-sync.pid  &>/dev/null && { echo "$BIN is already running"; exit 1; }
echo $$ > /run/btrfs-sync.pid

${DST_CMD[@]} "pgrep -f btrfs\ receive &>/dev/null" && { echo "btrfs-sync already running at destination"; exit 1; }

## src checks
echov "* Check source"
while read entry; do SRCS+=( "$entry" ); done < <(
  "${SRC_CMD[@]}" "
    for s in "${SRC[@]}"; do
      src=\"\$(cd \"\$s\" &>/dev/null && pwd)\" || { echo \"\$s not found\"; exit 1; } #abspath
      btrfs subvolume show \"\$src\" &>/dev/null && echo \"0|\$src\" || \
      for dir in \$( ls -drt \"\$src\"/* 2>/dev/null ); do
        DATE=\"\$( btrfs su sh \"\$dir\" 2>/dev/null | grep \"Creation time:\" | awk '{ print \$3, \$4 }' )\" \
        || continue   # not a subvolume
        SECS=\$( date -d \"\$DATE\" +\"%s\" )
        echo \"\$SECS|\$dir\"
      done
    done | sort -V | sed 's=.*|=='
  "
)
[[ ${#SRCS[@]} -eq 0 ]] && { echo "no BTRFS subvolumes found"; exit 1; }

## check pbzip2
[[ "$ZIP" == "pbzip2" ]] && {
    "${SRC_CMD[@]}" type pbzip2 &>/dev/null && \
    "${DST_CMD[@]}" type pbzip2 &>/dev/null || {
      echo "INFO: 'pbzip2' not installed on both ends, fallback to 'xz'"
      ZIP=xz PIZ=unxz
  }
}

## use 'pv' command if available
PV=( pv -F"time elapsed [%t] | rate %r | total size [%b]" )
[[ "$QUIET" == "1" ]] && PV=( cat ) || type pv &>/dev/null || {
  echo "INFO: install the 'pv' package in order to get a progress indicator"
  PV=( cat )
}

#----------------------------------------------------------------------------------------------------------

# sync snapshots

get_dst_snapshots() {      # sets DSTS DST_UUIDS
  local DST="$1"
  unset DSTS DST_UUIDS
  while read entry; do
    DST_UUIDS+=( "$( sed 's=|.*==' <<<"$entry" )" )
    DSTS+=(      "$( sed 's=.*|==' <<<"$entry" )" )
  done < <(
    "${DST_CMD[@]}" "
      DSTS=( \$( ls -d \"$DST\"/* 2>/dev/null ) )
      for dst in \${DSTS[@]}; do
        UUID=\$( sudo btrfs su sh \"\$dst\" 2>/dev/null | grep 'Received UUID' | awk '{ print \$3 }' )
        [[ \"\$UUID\" == \"-\" ]] || [[ \"\$UUID\" == \"\" ]] && continue
        echo \"\$UUID|\$dst\"
      done"
  )
}

choose_seed() {      # sets SEED
  local SRC="$1"

  SEED="$SEED_NEXT"
  if [[ "$SEED" == "" ]]; then
    # try to get most recent src snapshot that exists in dst to use as a seed
    local RXID_CALCULATED=0
    declare -A PATH_RXID DATE_RXID SHOWP RXIDP DATEP
    local LIST="$( "${SRC_CMD[@]}" sudo btrfs subvolume list -su "$SRC" )"
    SEED=$(
      for id in "${DST_UUIDS[@]}"; do
        # try to match by UUID
        local PATH_=$( awk "{ if ( \$14 == \"$id\" ) print \$16       }" <<<"$LIST" )
        local DATE=$(  awk "{ if ( \$14 == \"$id\" ) print \$11, \$12 }" <<<"$LIST" )

        # try to match by received UUID, only if necessary
        [[ "$PATH_" == "" ]] && {
          [[ "$RXID_CALCULATED" == "0" ]] && { # create table during the first iteration if needed
            local PATHS=( $( "${SRC_CMD[@]}" sudo btrfs su list -u "$SRC" | awk '{ print $11 }' ) )
            for p in "${PATHS[@]}"; do
              SHOWP="$( "${SRC_CMD[@]}" sudo btrfs su sh "$( dirname "$SRC" )/$( basename "$p" )" 2>/dev/null )"
              RXIDP="$( grep 'Received UUID' <<<"$SHOWP" | awk '{ print $3     }' )"
              DATEP="$( grep 'Creation time' <<<"$SHOWP" | awk '{ print $3, $4 }' )"
              [[ "$RXIDP" == "" ]] && continue
              PATH_RXID["$RXIDP"]="$p"
              DATE_RXID["$RXIDP"]="$DATEP"
            done
            RXID_CALCULATED=1
          }
          PATH_="${PATH_RXID["$id"]}"
           DATE="${DATE_RXID["$id"]}"
        }

        [[ "$PATH_" == "" ]] || [[ "$PATH_" == "$( basename "$SRC" )" ]] && continue

        local SECS=$( date -d "$DATE" +"%s" )
        echo "$SECS|$PATH_"
      done | sort -V | tail -1 | cut -f2 -d'|'
    )
  fi
}

exists_at_dst() {
  local SHOW="$( "${SRC_CMD[@]}" sudo btrfs subvolume show "$SRC" )"

  local SRC_UUID="$( grep 'UUID:' <<<"$SHOW" | head -1 | awk '{ print $2 }' )"
  grep -q "$SRC_UUID" <<<"${DST_UUIDS[@]}" && return 0;

  local SRC_RXID="$( grep 'Received UUID' <<<"$SHOW"   | awk '{ print $3 }' )"
  grep -q "^-$"       <<<"$SRC_RXID"       && return 1;
  grep -q "$SRC_RXID" <<<"${DST_UUIDS[@]}" && return 0;

  return 1
}

## sync incrementally
sync_snapshot() {
  local SRC="$1"
  "${SRC_CMD[@]}" test -d "$SRC" || return

  exists_at_dst "$SRC" && { echov "* Skip existing '$SRC'"; return 0; }

  choose_seed "$SRC"  # sets SEED

  # incremental sync argument
  [[ "$SEED" != "" ]] && {
    local SEED_PATH="$( dirname "$SRC" )/$( basename $SEED )"
    "${SRC_CMD[@]}" test -d "$SEED_PATH" &&
      local SEED_ARG=( -p "$SEED_PATH" ) || \
      echo "INFO: couldn't find $SEED_PATH. Non-incremental mode"
  }

  # do it
  echo -n "* Synchronizing '$src'"
  [[ "$SEED_ARG" != "" ]] && echov -n " using seed '$SEED'"
  echo "..."

  "${SRC_CMD[@]}" \
  sudo btrfs send -q ${SEED_ARG[@]} "$SRC" \
    | "$ZIP" \
    | "${PV[@]}" \
    | "${DST_CMD[@]}" "${PIZ[@]} | sudo btrfs receive \"$DST\" 2>&1 |(grep -v -e'^At subvol ' -e'^At snapshot '||true)" \
    || {
      "${DST_CMD[@]}" sudo btrfs subvolume delete "$DST"/"$( basename "$SRC" )" 2>/dev/null
      return 1;
    }

  # update DST list
  DSTS+=("$DST/$( basename "$SRC" )")
  DST_UUIDS+=("$SRC_UUID")
  SEED_NEXT="$SRC"
}

#----------------------------------------------------------------------------------------------------------

# sync all snapshots found in src
echov "* Check destination"
get_dst_snapshots "$DST" # sets DSTS DST_UUIDS
for src in "${SRCS[@]}"; do
  sync_snapshot "$src" && RET=0 || RET=1
  for i in $(seq 1 2); do
    [[ "$RET" != "1" ]] && break
    echo "* Retrying '$src'..."
    sync_snapshot "$src" && RET=0 || RET=1
  done
  [[ "$RET" == "1" ]] && { echo "Abort"; exit 1; }
done

#----------------------------------------------------------------------------------------------------------

# retention policy
[[ "$KEEP" != 0 ]] && \
  [[ ${#DSTS[@]} -gt $KEEP ]] && \
  echov "* Pruning old snapshots..." && \
  for (( i=0; i < $(( ${#DSTS[@]} - KEEP )); i++ )); do
    PRUNE_LIST+=( "${DSTS[$i]}" )
  done && \
  ${DST_CMD[@]} sudo btrfs subvolume delete "${PRUNE_LIST[@]}" $SILENT

# delete flag
[[ "$DELETE" == 1 ]] && \
  for dst in "${DSTS[@]}"; do
    FOUND=0
    for src in "${SRCS[@]}"; do
      [[ "$( basename $src )" == "$( basename $dst )" ]] && { FOUND=1; break; }
    done
    [[ "$FOUND" == 0 ]] && DEL_LIST+=( "$dst" )
  done
[[ "$DEL_LIST" != "" ]] && \
  echov "* Deleting non existent snapshots..." && \
  ${DST_CMD[@]} sudo btrfs subvolume delete "${DEL_LIST[@]}" $SILENT

exit 0

# License
#
# This script is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This script is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this script; if not, write to the
# Free Software Foundation, Inc., 59 Temple Place, Suite 330,


#!/bin/bash

#
# Script that creates BTRFS snapshots, manually or from cron
#
# Usage:
#          sudo btrfs-snp  <dir> (<tag>) (<limit>) (<seconds>) (<destdir>)
#
# Copyleft 2017 by Ignacio Nunez Hernanz <nacho _a_t_ ownyourbits _d_o_t_ com>
# GPL licensed (see end of file) * Use at your own risk!
#
# Based on btrfs-snap by Birger Monsen
#
# More at https://ownyourbits.com
#

function btrfs-snp()
{
  local   BIN="${0##*/}"
  local   DIR="${1}"
  local   TAG="${2:-snapshot}"
  local LIMIT="${3:-0}"
  local  TIME="${4:-0}"
  local   DST="${5:-.snapshots}"
  local MARGIN=15 # allow for some seconds of inaccuracy for cron / systemd timers

  ## usage
  [[ "$*" == "" ]] || [[ "$1" == "-h" ]] || [[ "$1" == "--help" ]] && {
    echo "Usage: $BIN <dir> (<tag>) (<limit>) (<seconds>) (<destdir>)

  dir     │ create snapshot of <dir>
  tag     │ name the snapshot <tag>_<timestamp>
  limit   │ keep <limit> snapshots with this tag. 0 to disable
  seconds │ don't create snapshots before <seconds> have passed from last with this tag. 0 to disable
  destdir │ store snapshot in <destdir>, path absolute or relative to <dir>

Cron example: Hourly snapshot for one day, daily for one week, weekly for one month, and monthly for one year.

cat > /etc/cron.hourly/$BIN <<EOF
#!/bin/bash
/usr/local/sbin/$BIN /home hourly  24 3600
/usr/local/sbin/$BIN /home daily    7 86400
/usr/local/sbin/$BIN /home weekly   4 604800
/usr/local/sbin/$BIN /     weekly   4 604800
/usr/local/sbin/$BIN /home monthly 12 2592000
EOF
chmod +x /etc/cron.hourly/$BIN"
    return 0
  }

  ## checks
  local SNAPSHOT=${TAG}_$( date +%F_%H%M%S )

  [[ ${EUID} -ne 0  ]] && { echo "Must be run as root. Try 'sudo $BIN'"; return 1; }
  [[ -d "$SNAPSHOT" ]] && { echo "$SNAPSHOT already exists"            ; return 1; }

  mount -t btrfs | cut -d' ' -f3 | grep -q "^${DIR}$" || {
    btrfs subvolume show "$DIR" &>/dev/null || {
      echo "$DIR is not a BTRFS mountpoint or snapshot"
      return 1
    }
  }

  [[ "$DST" = /* ]] || DST="$DIR/$DST"
  mkdir -p "$DST"
  local SNAPS=( $( ls -d "$DST/${TAG}_"* 2>/dev/null ) )

  ## check time of the last snapshot for this tag
  [[ "$TIME" != 0 ]] && [[ "${#SNAPS[@]}" != 0 ]] && {
    local LATEST=$( sed -r "s|.*_(.*_.*)|\\1|;s|_([0-9]{2})([0-9]{2})([0-9]{2})| \\1:\\2:\\3|" <<< "${SNAPS[-1]}" )
    LATEST=$( date +%s -d "$LATEST" ) || return 1

    [[ $(( LATEST + TIME )) -gt $(( $( date +%s ) + MARGIN )) ]] && { echo "No new snapshot needed for $TAG in $DIR"; return 0; }
  }

  ## do it
  btrfs subvolume snapshot -r "$DIR" "$DST/$SNAPSHOT" || return 1

  ## prune older backups
  [[ "$LIMIT" != 0 ]] && \
  [[ ${#SNAPS[@]} -ge $LIMIT ]] && \
    echo "Pruning old snapshots..." && \
    for (( i=0; i <= $(( ${#SNAPS[@]} - LIMIT )); i++ )); do
      btrfs subvolume delete "${SNAPS[$i]}"
    done

  echo "snapshot $SNAPSHOT generated"
}

btrfs-snp "$@"

# License
#
# This script is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This script is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this script; if not, write to the
# Free Software Foundation, Inc., 59 Temple Place, Suite 330,
# Boston, MA  02111-1307  USA

Install KACE Agent Remotely

KACE generally works best when installed as part of the initial setup process. If it’s included in the base image and that base image is rolled out to x number of devices, you know that the KACE agent will eventually check-in with your appliance. Then you’ll have a complete inventory of workstations.

Installed after-the-fact is when it gets a bit trickier. The preferred method is to have a GPO install it for you. This works fairly well if your users are 1) always connected to the network and 2) reboot from time to time and 3) the computers are in the correct OU (or the GPO is applied to the correct OU…). But this isn’t always the case – and even still there are instances in which the stars just don’t all align.

Then there is Covid/Remote work. Computer GPO and startup-based-User-GPOs just don’t work well with the current on-prem Domain Controllers and remote workforce. Ok, enough about this, let’s get onto how!

Enter PSExec, the tried and true remote management tool. This assumes you have administrative permissions to access the remote system AND that the remote system is somehow connected to your network (via VPN).

Remotely Connect to workstation
psexec \\computername powershell.exe
mkdir c:\kace
Copy KACE Agent to the remote workstation
cp \\yourkaceservername\client\agent_provisioning\windows_platform\ampagent-9.1.204-x86.msi c:\kace\
Run the MSI quietly
cd c:\kace
msiexec.exe /i ampagent-9.1.204-x86.msi host=fqdn.of.kace.server.tld nohooks=1 /qn
exit

If the computer is NOT running a recent version of powershell – looking at you Windows 7 – you’ll have to replace powershell.exe with cmd.exe. And since cmd.exe doesn’t support UNC paths you’ll have to use net use to mount the drives as a letter and then copy that way. Or just start > run > \\computername\c$, and manually copy to the c:\kace directory.

Another way is via the Windows Admin Center (https://docs.microsoft.com/en-us/windows-server/manage/windows-admin-center/deploy/install), Add the computer with required credentials, then launch PowerShell on the left-side panel.

I also found a few workstations that had a fully configured KACE agent installed but just refused to collect and send inventory information. In that case I ran a manual check:

c:\program files (x86)\quest\kace\runkbot 1 0
c:\program files (x86)\quest\kace\runkbot 4 0

List Ubuntu Version

When logging into a system it generally would show you the current version in a MOTD style window. This server had the MOTD changed so I needed to grab the pertinent information.

lsb_release -d
cat /etc/issue

Or on newer systems (16.04 or later)

cat /etc/os-release
hostnamectl

Remove Hidden KACE Agent

We utilize KACE (SMA K1000) for our helpdesk/ticketing, and inventory management. Part of our deployment packages include the KACE agent – this agent is required to send back data about the system it is installed upon (username, OS specs, hardware specs, etc).

Unfortunately, there are some times that the KACE agent doesn’t play nicely and it needs to be reinstalled. Or, in my case (pun intended), I needed to make a new GM image for deployment and it is recommended to NOT have the KACE agent installed prior to sysprep.

Open an administrator command prompt:

wmic product where "name like '%kace%'" call uninstall /nointeractive

This will find and remove any “KACE” related software currently residing on your system. You should see the messages “Method execution successful” and “ReturnValue = 0” if this runs successfully.

Note: This does not remove any existing firewall rules or files created outside of the standard install/uninstall configuration.

List Members of Dynamic Distribution Group

Like all good companies, we have dynamic distribution groups / lists that are based on the location of the user. Example, a user has the Office location of “Australia”, the dynamic list has the filter of:

((((((Office -eq 'Australia') -and (RecipientType -eq 'UserMailbox'))) -or (((CustomAttribute2 -like 'IncludeAllAUSLists') -and (-not(CustomAttribute1 -like 'excludefromdynamic')))))) -and (-not(Name -like 'SystemMailbox{*')) -and (-not(Name -like 'CAS_{*')) -and (-not(RecipientTypeDetailsValue -eq 'MailboxPlan')) -and (-not(RecipientTypeDetailsValue -eq 'DiscoveryMailbox')) -and (-not(RecipientTypeDetailsValue -eq 'PublicFolderMailbox')) -and (-not(RecipientTypeDetailsValue -eq 'ArbitrationMailbox')) -and (-not(RecipientTypeDetailsValue -eq 'AuditLogMailbox')) -and (-not(RecipientTypeDetailsValue -eq 'AuxAuditLogMailbox')) -and (-not(RecipientTypeDetailsValue -eq 'SupervisoryReviewPolicyMailbox')))

The user was added and then the office location was of Australia was added (after troubleshooting below). He complained he did not receive an email that was sent to this distribution list, so I had to verify he was a member.

Connect to your Office 365 online powershell (I have it previously documented if you don’t remember).

$GROUP = Get-DynamicDistributionGroup "All-Australia-Employees"
Get-Recipient -RecipientPreviewFilter $FTE.RecipientFilter -OrganizationalUnit $FTE.RecipientContainer

File Folder Permissions

Had a user with full modify permissions to some directories on a file server. They managed to completely fubar up the permissions when trying to 1) lock it down to just them and 2) disallow other users. You can see where this is going.

The backup user and local administrator no longer had access. I attempted to take ownership using the GUI, but it would just fail saying I don’t have permissions. I even “ran as administrator” to no avail.

Command line to the rescue!

  • Open Command Prompt as an Administrator
  • takeown /f “directory\subdir\moredir\*.*” /r /d y

Ramblings Of An IT Person